{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# pytorch实现自动求导autograd\n",
    "torch.autograd就是为方便用户使用，而专门开发的一套自动求导引擎，它能够根据输入和前向传播过程自动构建计算图，并执行反向传播\n",
    "\n",
    "计算图(Computation Graph)是现代深度学习框架如PyTorch和TensorFlow等的核心，其为高效自动求导算法——反向传播(Back Propogation)提供了理论支持，了解计算图在实际写程序过程中会有极大的帮助。计算图的相关内容在吴恩达深度学习课程和邱希鹏《神经网络与深度学习》中，均有讲到，可以看我之前的[博客](https://xiuzhedorothy.gitee.io/2020/01/19/qian-kui-shen-jing-wang-luo-xiao-jie/ \"博客\")😀\n",
    "![](https://s2.ax1x.com/2020/01/19/1CNLo6.png)\n",
    "## 写在前面\n",
    "不知道是因为这一章的东西比较硬核还是怎么的，学的时候半懂不懂的，不过学完之后总结到了方法：这一块不懂可以接着往下看，因为后面还会着重再讲这些东西，比如variable.backward()的三个参数，看半天看不懂，但后面会慢慢逐个说明👨‍🎓\n",
    "## requires_grad\n",
    "autograd中的核心数据结构是Variable。从v0.4版本起，Variable和Tensor合并。可以认为**需要求导的tensor**(requires_grad)即Variable。\n",
    "\n",
    "Variable提供了大部分tensor支持的函数，但不支持部分inplace函数，因为它们会修改tensor自身，而在反向传播中，variable需要缓存原来的tensor来计算反向传播梯度。计算各个Variable的梯度，调用它们根节点Variable的`backward`方法即可，autograd会自动沿着计算图反向传播，计算每一个叶子节点的梯度(下来会举例解释)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pre\n",
    "import torch as t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`variable.backward(gradient=None, retain_graph=None, create_graph=None)`的参数解释：\n",
    "\n",
    "- gradient：有关gradient的介绍见后面（书上的好像不太对，反正看不懂）\n",
    "- retain_graph：反向传播需要缓存一些中间结果，反向传播之后，这些缓存就被清空，可通过指定这个参数不清空缓存，用来多次反向传播。\n",
    "- create_graph：构建计算图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1.6719e-01, -6.9152e-01, -1.9536e+00, -1.8011e-02],\n",
      "        [-1.6282e+00, -1.1404e-03, -1.2241e+00, -1.1264e+00],\n",
      "        [-1.5771e+00,  1.5322e+00,  1.6548e+00, -1.5076e+00]],\n",
      "       requires_grad=True)\n",
      "tensor([[0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0.]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "a=t.randn(3,4,requires_grad=True)\n",
    "'''\n",
    "其他形式：\n",
    "    a=t.randn(3,4).requires_grad_()\n",
    "\n",
    "    a=t.randn(3,4)\n",
    "    a.requires_grad=True\n",
    "'''\n",
    "print(a)\n",
    "b=t.zeros(3,4).requires_grad_()\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1.6719e-01, -6.9152e-01, -1.9536e+00, -1.8011e-02],\n",
      "        [-1.6282e+00, -1.1404e-03, -1.2241e+00, -1.1264e+00],\n",
      "        [-1.5771e+00,  1.5322e+00,  1.6548e+00, -1.5076e+00]],\n",
      "       grad_fn=<AddBackward0>)\n",
      "tensor(-6.3735, grad_fn=<SumBackward0>)\n",
      "None\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "c=a+b\n",
    "print(c)\n",
    "d = c.sum()\n",
    "print(d)\n",
    "print(d.backward()) # 反向传播\n",
    "print(d.requires_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.]])\n",
      "True True True\n"
     ]
    }
   ],
   "source": [
    "print(a.grad) # 即对a求导\n",
    "\n",
    "# 此处虽然没有指定c需要求导，但c依赖于a，而a需要求导，\n",
    "# 因此c的requires_grad属性会自动设为True\n",
    "print(a.requires_grad, b.requires_grad, c.requires_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True True False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# 由用户创建的variable属于叶子节点，对应的grad_fn是None\n",
    "print(a.is_leaf, b.is_leaf, c.is_leaf)\n",
    "\n",
    "# c.grad是None, 因c不是叶子节点，它的梯度是用来计算a的梯度\n",
    "# 所以虽然c.requires_grad = True,但其梯度计算完之后即被释放\n",
    "print(c.grad is None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面写了这么多，其实我一个都没看懂。。。😥，感觉讲的怪怪的，一会冒出来一个参数一会儿又冒出来一个参数，这个例子感觉举的也不好，于是从apachecn的pytorch文档又扒了一个例子再来试一遍。\n",
    "    PyTorch中，所有神经网络的核心是`autograd`包，`autograd`包中的**核心类**是`torch.Tensor`，如果将生成的tensor的属性设置为`.requires_grad=True`，如下面的例子：\n",
    " ```python\n",
    "a=t.randn(3,4,requires_grad=True)\n",
    "```\n",
    "那么它将会追踪对于该张量的所有操作,当完成计算后：\n",
    "- `backward()`方法可以自动计算所有梯度\n",
    "- 这个张量的所有梯度将会自动累加到`.grad`属性\n",
    "\n",
    "要阻止一个张量被跟踪历史，可以调用` .detach()`方法将其与计算历史分离，并阻止它未来的计算记录被跟踪\n",
    "\n",
    "为了防止跟踪历史记录（和使用内存），可以将代码块包装在 with torch.no_grad(): 中。在**评估模型**时特别有用，因为模型可能具有 requires_grad = True 的可训练的参数，但是评估过程并不需要计算梯度。\n",
    "\n",
    "除了上文提到的`torch.Tensor`类，还有一个类：`Function`对autograd的实现非常重要\n",
    "\n",
    "- 每个张量都有一个`.grad_fn`属性，该属性引用了创建`Tensor`自身的`Function`（除非这个张量是用户手动创建的，即这个张量的`grad_fn`是`None`）\n",
    "- `Tensor`和`Function`互相连接生成了一个**无环图(acyclic graph)**\n",
    "\n",
    "> 在图论中，如果一个有向图从任意顶点出发无法经过若干条边回到该点，则这个图是一个**有向无环图（DAG,directed acyclic graph）**\n",
    "\n",
    "如果需要计算导数，可以在`Tensor`上调用`.backward()`。如果`Tensor`是一个标量（即它包含一个元素的数据），则不需要为`backward()`指定任何参数，但是如果它有更多的元素，则需要指定一个`gradient`参数，该参数是形状匹配的张量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "x=t.ones(2,2,requires_grad=True)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[3., 3.],\n",
      "        [3., 3.]], grad_fn=<AddBackward0>)\n",
      "\n",
      " <AddBackward0 object at 0x000001E2ECA32F98>\n"
     ]
    }
   ],
   "source": [
    "y=x+2\n",
    "print(y)\n",
    "print('\\n',y.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的结果可见y有`grad_fn`属性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[27., 27.],\n",
      "        [27., 27.]], grad_fn=<MulBackward0>) tensor(27., grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "source": [
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "print(z, out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[4.5000, 4.5000],\n",
      "        [4.5000, 4.5000]])\n"
     ]
    }
   ],
   "source": [
    "out.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的计算过程的公式为：\n",
    "$y=x+2$，\n",
    "$z=3y^{2}$，\n",
    "$out=\\frac{1}{n}\\sum_{i=1}^{n}z_{i}$\n",
    "\n",
    "`x.grad`即导数$\\frac{\\partial(out)}{\\partial x}$\n",
    "\n",
    "$out=\\frac{1}{4}\\sum_{i}^{4}z_{i}$，$z_{i}=3(x_{i}+2)^{2}$，所以$z_{i}|_{x_{i}=1}=27$，因此有：\n",
    "\n",
    "$\\frac{\\partial(out)}{\\partial x_{i}}=\\frac{3}{2}(x_{i}+2)$，代入得 $4.5$\n",
    "\n",
    "---------\n",
    "数学上，若有向量值函数 $$\\vec{y}=f(\\vec{x})$$，那么 $$\\vec{y}$$ 相对于 $$\\vec{x}$$的梯度是一个雅可比矩阵：\n",
    "\n",
    "$$\n",
    "J=\\left(\\begin{array}{ccc}\n",
    "   \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{1}}\\\\\n",
    "   \\vdots & \\ddots & \\vdots\\\\\n",
    "   \\frac{\\partial y_{1}}{\\partial x_{n}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
    "   \\end{array}\\right)\n",
    "$$\n",
    "\n",
    "通常来说，`torch.autograd` 是计算雅可比向量积的一个“引擎”。也就是说，给定任意向量 $$v=\\left(\\begin{array}{cccc} v_{1} & v_{2} & \\cdots & v_{m}\\end{array}\\right)^{T}$$，计算乘积 $$v^{T}\\cdot J$$。如果 $$v$$ 恰好是一个标量函数 $$l=g\\left(\\vec{y}\\right)$$ 的导数，即 $$v=\\left(\\begin{array}{ccc}\\frac{\\partial l}{\\partial y_{1}} & \\cdots & \\frac{\\partial l}{\\partial y_{m}}\\end{array}\\right)^{T}$$，那么根据链式法则，雅可比向量积应该是 $$l$$ 对 $$\\vec{x}$$ 的导数：\n",
    "\n",
    "$$\n",
    "J^{T}\\cdot v=\\left(\\begin{array}{ccc}\n",
    "   \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{1}}\\\\\n",
    "   \\vdots & \\ddots & \\vdots\\\\\n",
    "   \\frac{\\partial y_{1}}{\\partial x_{n}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
    "   \\end{array}\\right)\\left(\\begin{array}{c}\n",
    "   \\frac{\\partial l}{\\partial y_{1}}\\\\\n",
    "   \\vdots\\\\\n",
    "   \\frac{\\partial l}{\\partial y_{m}}\n",
    "   \\end{array}\\right)=\\left(\\begin{array}{c}\n",
    "   \\frac{\\partial l}{\\partial x_{1}}\\\\\n",
    "   \\vdots\\\\\n",
    "   \\frac{\\partial l}{\\partial x_{n}}\n",
    "   \\end{array}\\right)\n",
    "$$\n",
    "\n",
    "（注意：行向量的$$ v^{T}\\cdot J$$也可以被视作列向量的$$J^{T}\\cdot v$$)\n",
    "\n",
    "雅可比向量积的这一特性使得将外部梯度输入到具有非标量输出的模型中变得非常方便。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "**再来看一个例子：**\n",
    "\n",
    "计算下面这个函数的导函数：\n",
    "$$\n",
    "y = x^2\\bullet e^x\n",
    "$$\n",
    "它的导函数是：\n",
    "$$\n",
    "{dy \\over dx} = 2x\\bullet e^x + x^2 \\bullet e^x\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[4.8295e-02, 4.9038e-01, 1.4586e+00, 4.8821e-02],\n",
       "        [2.2538e-02, 7.9433e-03, 1.1186e-02, 9.6326e-03],\n",
       "        [4.0448e+01, 2.8294e-01, 1.5485e-01, 6.9728e-01]],\n",
       "       grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(x):\n",
    "    y=x**2*t.exp(x)\n",
    "    return y\n",
    "x = t.randn(3,4, requires_grad = True)\n",
    "y = f(x)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.3398,  2.3211,  5.0744,  0.5372],\n",
      "        [-0.2542,  0.1940, -0.1888,  0.2153],\n",
      "        [77.9019,  1.6014,  1.0845,  2.9677]])\n"
     ]
    }
   ],
   "source": [
    "y.backward(t.ones(y.size()))\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算图\n",
    "PyTorch中`autograd`的底层采用了计算图，计算图是一种特殊的有向无环图（DAG），用于记录算子与变量之间的关系。一般用矩形表示算子，椭圆形表示变量。如表达式$ \\textbf {z = wx + b}$可分解为$\\textbf{y = wx}$和$\\textbf{z = y + b}$，其计算图如图3-3所示，图中`MUL`，`ADD`都是算子，$\\textbf{w}$，$\\textbf{x}$，$\\textbf{b}$即变量。\n",
    "\n",
    "![](https://s2.ax1x.com/2020/01/31/11qx1I.png)\n",
    "\n",
    "如上有向无环图中，$\\textbf{X}$和$\\textbf{b}$是叶子节点（leaf node），这些节点通常由用户自己创建，不依赖于其他变量。$\\textbf{z}$称为根节点，是计算图的最终目标。利用链式法则很容易求得各个叶子节点的梯度。\n",
    "$${\\partial z \\over \\partial b} = 1,\\space {\\partial z \\over \\partial y} = 1\\\\\n",
    "{\\partial y \\over \\partial w }= x,{\\partial y \\over \\partial x}= w\\\\\n",
    "{\\partial z \\over \\partial x}= {\\partial z \\over \\partial y} {\\partial y \\over \\partial x}=1 * w\\\\\n",
    "{\\partial z \\over \\partial w}= {\\partial z \\over \\partial y} {\\partial y \\over \\partial w}=1 * x\\\\\n",
    "$$\n",
    "而有了计算图，上述链式求导即可利用计算图的反向传播自动完成，如下图\n",
    "![](https://s2.ax1x.com/2020/01/31/11LaDK.png)\n",
    "代码细节："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False True True\n",
      "True\n",
      "True True True\n",
      "False False\n",
      "\n",
      " <AddBackward0 object at 0x000001E2ECA63908>\n",
      "\n",
      " 输出一个元组： ((<MulBackward0 object at 0x000001E2ECA638D0>, 0), (<AccumulateGrad object at 0x000001E2ECA63978>, 0))\n"
     ]
    }
   ],
   "source": [
    "x = t.ones(1)\n",
    "b = t.rand(1, requires_grad = True)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "y = w * x # 等价于y=w.mul(x)\n",
    "z = y + b # 等价于z=y.add(b)\n",
    "print(x.requires_grad, b.requires_grad, w.requires_grad)\n",
    "print(y.requires_grad)\n",
    "print(x.is_leaf, w.is_leaf, b.is_leaf)\n",
    "print(y.is_leaf, z.is_leaf)\n",
    "\n",
    "# grad_fn可以查看这个variable的反向传播函数，\n",
    "# z是add函数的输出，所以它的反向传播函数是AddBackward\n",
    "print('\\n',z.grad_fn)\n",
    "\n",
    "# next_functions保存grad_fn的输入，是一个tuple，tuple的元素也是Function\n",
    "# 第一个是y，它是乘法(mul)的输出，所以对应的反向传播函数y.grad_fn是MulBackward\n",
    "# 第二个是b，它是叶子节点，由用户创建\n",
    "print('\\n','输出一个元组：',z.grad_fn.next_functions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True \n",
      "\n",
      "[ 0 0 ] <MulBackward0 object at 0x000001E2ECA64438>\n",
      "[ 0 1 ] 0\n",
      "[ 1 0 ] <AccumulateGrad object at 0x000001E2ECA645C0>\n",
      "[ 1 1 ] 0\n"
     ]
    }
   ],
   "source": [
    "# variable的grad_fn对应着和图中的function相对应\n",
    "print(z.grad_fn.next_functions[0][0] == y.grad_fn,'\\n')\n",
    "\n",
    "# 注意元组的输出各种输出↓\n",
    "for i in range(2):\n",
    "    for ii in range(2):\n",
    "        print('[',i,ii,']',z.grad_fn.next_functions[i][ii])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((<AccumulateGrad at 0x1e2eca49b70>, 0), (None, 0))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 第一个是w，叶子节点，需要求导，梯度是累加的\n",
    "# 第二个是x，叶子节点，不需要求导，所以为None\n",
    "y.grad_fn.next_functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, None)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 叶子节点的grad_fn是None\n",
    "w.grad_fn,x.grad_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算w的梯度的时候，需要用到x的数值(${\\partial y\\over \\partial w} = x $)，这些数值在前向过程中会保存成buffer，在计算完梯度之后会自动清空。为了能够多次反向传播需要指定`retain_graph`来保留这些buffer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1.])\n",
      "tensor([2.])\n",
      "tensor([3.])\n",
      "tensor([4.])\n",
      "tensor([5.])\n"
     ]
    }
   ],
   "source": [
    "# 使用retain_graph来保存buffer\n",
    "for i in range(5):\n",
    "    z.backward(retain_graph=True)\n",
    "    print(w.grad)\n",
    "# 多次反向传播，梯度累加，这也就是w中AccumulateGrad标识的含义"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 6., 3., 2.])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(x):\n",
    "    result = 1\n",
    "    for ii in x:\n",
    "        if ii.item()>0: \n",
    "            result=ii*result\n",
    "    return result\n",
    "x = t.arange(-2,4,dtype=t.float32).requires_grad_()\n",
    "y = f(x) # y = x[3]*x[4]*x[5]\n",
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-2., -1.,  0.,  1.,  2.,  3.], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有些时候我们可能不希望autograd对tensor求导。认为求导需要缓存许多中间结构，增加额外的内存/显存开销，那么我们可以关闭自动求导。对于不需要反向传播的情景（如inference，即测试推理时），关闭自动求导可实现一定程度的速度提升，并节省约一半显存，因其不需要分配空间计算梯度。**下面便是一个利用with函数关闭自动求梯度的代码：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True, False)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with t.no_grad():\n",
    "    x = t.ones(1)\n",
    "    w = t.rand(1, requires_grad = True)\n",
    "    y = x * w\n",
    "# y依赖于w和x，虽然w.requires_grad = True，但是y的requires_grad依旧为False\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch.autograd.grad_mode.set_grad_enabled at 0x1e2eca67048>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 恢复默认配置\n",
    "t.set_grad_enabled(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果想要修改tensor的数值，但是又不希望被autograd记录，那么可以对tensor.data进行操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.]])\n",
      "False\n",
      "是否自动计算梯度： False \n",
      "\n",
      "tensor([[0.7311, 0.7311, 0.7311, 0.7311],\n",
      "        [0.7311, 0.7311, 0.7311, 0.7311],\n",
      "        [0.7311, 0.7311, 0.7311, 0.7311]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "a = t.ones(3,4,requires_grad=True)\n",
    "b = t.ones(3,4,requires_grad=True)\n",
    "c = a * b\n",
    "\n",
    "print(a.data) # 还是一个tensor\n",
    "print(a.data.requires_grad) # 但是已经是独立于计算图之外\n",
    "\n",
    "d = a.data.sigmoid_() # sigmoid_ 是个inplace操作，会修改a自身的值\n",
    "print('是否自动计算梯度：',d.requires_grad,'\\n')\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们希望对tensor进行操作，但是又不希望被记录, 可以使用tensor.data 或者tensor.detach()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 近似于 tensor=a.data, 但是如果tensor被修改，backward可能会报错\n",
    "tensor = a.detach()\n",
    "tensor.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 统计tensor的一些指标，不希望被记录\n",
    "mean = tensor.mean()\n",
    "std = tensor.std()\n",
    "maximum = tensor.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor[0]=1\n",
    "# 下面会报错：　RuntimeError: one of the variables needed for gradient\n",
    "#             computation has been modified by an inplace operation\n",
    "#　因为 c=a*b, b的梯度取决于a，现在修改了tensor，其实也就是修改了a，梯度不再准确\n",
    "# c.sum().backward() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在反向传播过程中非叶子节点的导数计算完之后即被清空。若想查看这些变量的梯度，有两种方法：\n",
    "- 使用autograd.grad函数\n",
    "- 使用hook\n",
    "\n",
    "`autograd.grad`和`hook`方法都是很强大的工具，更详细的用法参考官方api文档，这里举例说明基础的使用。推荐使用`hook`方法，但是在实际使用中应尽量避免修改grad的值。\n",
    "\n",
    "**代码略（以后要用的时候再看，现在只看不用等于白学）**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">最后再来看看variable中grad属性和backward函数`grad_variables`参数的含义，这里直接下结论：\n",
    "- variable $\\textbf{x}$的梯度是目标函数${f(x)} $对$\\textbf{x}$的梯度，$\\frac{df(x)}{dx} = (\\frac {df(x)}{dx_0},\\frac {df(x)}{dx_1},...,\\frac {df(x)}{dx_N})$，形状和$\\textbf{x}$一致。\n",
    "- 对于y.backward(grad_variables)中的grad_variables相当于链式求导法则中的$\\frac{\\partial z}{\\partial x} = \\frac{\\partial z}{\\partial y} \\frac{\\partial y}{\\partial x}$中的$\\frac{\\partial z}{\\partial y}$。z是目标函数，一般是一个标量，故而$\\frac{\\partial z}{\\partial y}$的形状与variable $\\textbf{y}$的形状一致。`z.backward()`在一定程度上等价于y.backward(grad_y)。`z.backward()`省略了grad_variables参数，是因为$z$是一个标量，而$\\frac{\\partial z}{\\partial z} = 1$\n",
    "\n",
    "以上是《深度学习框架PyTorch：入门与实践》中关于`grad_variables`参数 的介绍，但在我目前用的版本(1.2)，`grad_variables`被替换成了`gradient`参数。\n",
    "> 如果需要计算导数，可以在Tensor上调用.backward()。如果Tensor是一个标量（即它包含一个元素的数据），则不需要为backward()指定任何参数，但是如果它有更多的元素，则需要指定一个gradient参数，该参数是形状匹配的张量。(来自apachecn pytorch快速入门)\n",
    "\n",
    "<font color=red>pytorch 1.0中文文档：</font>\n",
    "- gradient (*Tensor 或 None*) – 关于张量的梯度。如果它是一个张量，它将被自动转换成不要求梯度的张量，除非`create_graph`是`True`。标量张量或不需要梯度的可用`None`指定。如果None对所有grad_tensors可接受，则此参数可选。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=red>在PyTorch中计算图的特点可总结如下：</font>\n",
    "\n",
    "- autograd根据用户对variable的操作构建其计算图。对变量的操作抽象为`Function`。\n",
    "- 对于那些不是任何函数(Function)的输出，由用户创建的节点称为叶子节点，叶子节点的`grad_fn`为None。叶子节点中需要求导的variable，具有`AccumulateGrad`标识，因其梯度是累加的。\n",
    "- variable默认是不需要求导的，即`requires_grad`属性默认为False，如果某一个节点requires_grad被设置为True，那么所有依赖它的节点`requires_grad`都为True。\n",
    "- variable的`volatile`属性默认为False，如果某一个variable的`volatile`属性被设为True，那么所有依赖它的节点`volatile`属性都为True。volatile属性为True的节点不会求导，volatile的优先级比`requires_grad`高。\n",
    "- **多次反向传播时，梯度是累加的**。反向传播的中间缓存会被清空，为进行多次反向传播需指定`retain_graph`=True来保存这些缓存。\n",
    "- 非叶子节点的梯度计算完之后即被清空，可以使用`autograd.grad`或`hook`技术获取非叶子节点的值。\n",
    "- variable的grad与data形状一致，应避免直接修改variable.data，因为对data的直接操作无法利用autograd进行反向传播\n",
    "- 反向传播函数`backward`的参数`grad_variables`可以看成链式求导的中间结果，如果是标量，可以省略，默认为1\n",
    "- PyTorch采用动态图设计，可以很方便地查看中间层的输出，动态的设计计算图结构。\n",
    "\n",
    "这些知识不懂大多数情况下也不会影响对pytorch的使用，但是掌握这些知识有助于更好的理解pytorch，并有效的避开很多陷阱"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 扩展autograd（暂略）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 用autograd实现线性回归"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch as t\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython import display \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_fake_data(batch_size=8):\n",
    "    ''' 产生随机数据：y = x*2 + 3，加上了一些噪声'''\n",
    "    x = t.rand(batch_size,1) * 5\n",
    "    y = x * 2 + 3 + t.randn(batch_size, 1)\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1e2ff715358>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAMy0lEQVR4nO3db2xdd33H8c8Hx4hbGDJT7ybsEAwSskDtwJOFulViUgtyGBW1Kh60UlG3VcqTDco0mdXiAeJRJ3lCTBoaikpXJKqyqTMG8QA3Aqo+KQUHF9ySGhDjT67ZYlRZ+8MVdcx3D3ydJRcnPvfeY5/zdd4vKYrvyY3PV0fJW0e/c+6xI0IAgHxeUfUAAID+EHAASIqAA0BSBBwAkiLgAJDUscPc2Y033hjj4+OHuUsASO/s2bO/jIhm9/ZDDfj4+LiWl5cPc5cAkJ7tn+61nSUUAEiKgANAUgQcAJIi4ACQFAEHgKQIOAAkdai3EQLA9WRxpaX5pTWtb7Y1OtLQ7PSEZibHSvv+BBwADsDiSktzC6tqb21Lklqbbc0trEpSaRFnCQUADsD80tqleO9qb21rfmmttH0QcAA4AOub7Z6292PfgNt+xPYF289ftm3e9ou2v2f7i7ZHSpsIAI6A0ZFGT9v7UeQM/FFJJ7u2nZF0U0T8gaQfSJorbSIAOAJmpyfUGB66YltjeEiz0xOl7WPfgEfE05Je6tr2ZERc7Lz8pqTjpU0EAEfAzOSYHrrrZo2NNGRJYyMNPXTXzbW7C+UvJP3L1f7Q9ilJpyTpxIkTJewOAHKYmRwrNdjdBrqIaftjki5Keuxq74mI0xExFRFTzeZvPc4WANCnvs/Abd8n6Q5Jt0dElDcSAKCIvgJu+6Skv5X0JxHxq3JHAgAUUeQ2wsclPSNpwvZ52/dL+kdJvyPpjO3nbH/mgOcEAHTZ9ww8Iu7ZY/NnD2AWAEAP+CQmACRFwAEgKQIOAEkRcABIioADQFIEHACSIuAAkBQBB4CkCDgAJEXAASApAg4ASRFwAEiKgANAUgQcAJIi4ACQFAEHgKQIOAAkRcABICkCDgBJEXAASIqAA0BSBBwAkiLgAJAUAQeApAg4ACRFwAEgKQIOAEkRcABIioADQFIEHACSIuAAkBQBB4CkCDgAJEXAASApAg4ASRFwAEhq34DbfsT2BdvPX7btd22fsf3Dzu+vO9gxAQDdipyBPyrpZNe2ByV9LSLeIulrndcAgEO0b8Aj4mlJL3VtvlPS5zpff07STMlzAQD20e8a+O9HxC8kqfP7713tjbZP2V62vbyxsdHn7gAA3Q78ImZEnI6IqYiYajabB707ALhu9Bvw/7T9eknq/H6hvJEAAEX0G/AvS7qv8/V9kr5UzjgAgKKK3Eb4uKRnJE3YPm/7fkl/J+k9tn8o6T2d1wCAQ3RsvzdExD1X+aPbS54FANADPokJAEkRcABIioADQFIEHACSIuAAkBQBB4CkCDgAJEXAASApAg4ASRFwAEiKgANAUgQcAJIi4ACQFAEHgKQIOAAkRcABIKl9f6ADgMOxuNLS/NKa1jfbGh1paHZ6QjOTY1WPhRoj4EANLK60NLewqvbWtiSptdnW3MKqJBFxXBVLKEANzC+tXYr3rvbWtuaX1iqaCBkQcKAG1jfbPW0HJAIO1MLoSKOn7YBEwIFamJ2eUGN46IptjeEhzU5PVDQRMuAiJlADuxcquQsFvSDgQE3MTI4RbPSEJRQASIqAA0BSBBwAkiLgAJAUAQeApAg4ACRFwAEgKe4DB2qIR8uiCAIO1AyPlkVRLKEANcOjZVEUZ+DAAepnKYRHy6Kogc7Abf+17RdsP2/7cduvKmswILvdpZDWZluh/18KWVxpXfPv8WhZFNV3wG2PSfqwpKmIuEnSkKS7yxoMyK7fpRAeLYuiBl1COSapYXtL0g2S1gcfCTga+l0K4dGyKKrvgEdEy/bfS/qZpLakJyPiye732T4l6ZQknThxot/dAemMjjTU2iPWRZZCeLQsihhkCeV1ku6U9CZJo5Jebfve7vdFxOmImIqIqWaz2f+kQDIsheCgDXIR892S/j0iNiJiS9KCpD8uZywgv5nJMT10180aG2nIksZGGnrorps5s0ZpBlkD/5mkW2zfoJ0llNslLZcyFXBEsBSCgzTIGviztp+Q9B1JFyWtSDpd1mBAXfExd9TFQHehRMTHJX28pFmA2uNj7qgTPkoP9ICPuaNOCDjQAz7mjjoh4EAP+Jg76oSAAz3g3m7UCU8jBHrAx9xRJwQc6BH3dqMuWEIBgKQIOAAkRcABICkCDgBJEXAASIqAA0BSBBwAkiLgAJAUAQeApAg4ACRFwAEgKQIOAEkRcABIioADQFIEHACS4nngKNXiSosfdgAcEgKO0iyutDS3sHrpp7a3NtuaW1iVJCIOHACWUFCa+aW1S/He1d7a1vzSWkUTAUcbAUdp1jfbPW0HMBgCjtKMjjR62g5gMAQcpZmdnlBjeOiKbY3hIc1OT1Q0EXC0cRETpdm9UMldKMDhIOAo1czkGMEGDglLKACQFAEHgKQIOAAkRcABICkCDgBJEXAASGqggNsesf2E7Rdtn7P9R2UNBgC4tkHvA/8HSV+NiA/YfqWkG0qYCQBQQN8Bt/1aSe+S9GeSFBEvS3q5nLEAAPsZZAnlzZI2JP2z7RXbD9t+dfebbJ+yvWx7eWNjY4DdAQAuN0jAj0n6Q0n/FBGTkv5X0oPdb4qI0xExFRFTzWZzgN0BAC43SMDPSzofEc92Xj+hnaADAA5B3wGPiP+Q9HPbu88KvV3S90uZCgCwr0HvQvmQpMc6d6D8WNKfDz4SAKCIgQIeEc9JmippFgBAD/gkJgAkRcABICkCDgBJ8SPVamBxpcXPkQTQMwJescWVluYWVtXe2pYktTbbmltYlSQiDuCaWEKp2PzS2qV472pvbWt+aa2iiQBkQcArtr7Z7mk7AOwi4BUbHWn0tB0AdhHwis1OT6gxPHTFtsbwkGanJ67yNwBgBxcxK7Z7oZK7UAD0ioDXwMzkGMEG0DOWUAAgKQIOAEkRcABIioADQFIEHACSIuAAkBQBB4CkCDgAJEXAASApAg4ASRFwAEiKgANAUgQcAJIi4ACQFAEHgKQIOAAkRcABICkCDgBJEXAASIqAA0BSBBwAkiLgAJAUAQeApAg4ACQ1cMBtD9lesf2VMgYCABRTxhn4A5LOlfB9AAA9GCjgto9Lep+kh8sZBwBQ1KBn4J+S9FFJvylhFgBAD/oOuO07JF2IiLP7vO+U7WXbyxsbG/3uDgDQZZAz8Fslvd/2TyR9QdJttj/f/aaIOB0RUxEx1Ww2B9gdAOByfQc8IuYi4nhEjEu6W9LXI+Le0iYDAFwT94EDQFLHyvgmEfGUpKfK+F4AgGI4AweApAg4ACRFwAEgKQIOAEkRcABIioADQFIEHACSKuU+8IO2uNLS/NKa1jfbGh1paHZ6QjOTY1WPBQCVqn3AF1damltYVXtrW5LU2mxrbmFVkog4gOta7ZdQ5pfWLsV7V3trW/NLaxVNBAD1UPuAr2+2e9oOANeL2gd8dKTR03YAuF7UPuCz0xNqDA9dsa0xPKTZ6YmKJgKAeqj9RczdC5XchQIAV6p9wKWdiBNsALhS7ZdQAAB7I+AAkBQBB4CkCDgAJEXAASApR8Th7czekPTTgm+/UdIvD3Cco4LjVAzHqRiOUzGHfZzeGBHN7o2HGvBe2F6OiKmq56g7jlMxHKdiOE7F1OU4sYQCAEkRcABIqs4BP131AElwnIrhOBXDcSqmFseptmvgAIBrq/MZOADgGgg4ACRVu4DbPml7zfaPbD9Y9Tx1ZfsR2xdsP1/1LHVm+w22v2H7nO0XbD9Q9Ux1ZPtVtr9l+7ud4/SJqmeqM9tDtldsf6XKOWoVcNtDkj4t6b2S3ibpHttvq3aq2npU0smqh0jgoqS/iYi3SrpF0l/yb2pPv5Z0W0S8XdI7JJ20fUvFM9XZA5LOVT1ErQIu6Z2SfhQRP46IlyV9QdKdFc9USxHxtKSXqp6j7iLiFxHxnc7X/62d/3Q8XL5L7Pifzsvhzi/ucNiD7eOS3ifp4apnqVvAxyT9/LLX58V/NpTE9rikSUnPVjtJPXWWBZ6TdEHSmYjgOO3tU5I+Kuk3VQ9St4B7j22cBWBgtl8j6d8kfSQi/qvqeeooIrYj4h2Sjkt6p+2bqp6pbmzfIelCRJytehapfgE/L+kNl70+Lmm9ollwRNge1k68H4uIharnqbuI2JT0lLjGspdbJb3f9k+0s8R7m+3PVzVM3QL+bUlvsf0m26+UdLekL1c8ExKzbUmflXQuIj5Z9Tx1Zbtpe6TzdUPSuyW9WO1U9RMRcxFxPCLGtdOnr0fEvVXNU6uAR8RFSX8laUk7F5v+NSJeqHaqerL9uKRnJE3YPm/7/qpnqqlbJX1QO2dKz3V+/WnVQ9XQ6yV9w/b3tHMidSYiKr1FDvvjo/QAkFStzsABAMURcABIioADQFIEHACSIuAAkBQBB4CkCDgAJPV/p31ox2pXvwYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x, y = get_fake_data()\n",
    "plt.scatter(x.squeeze().numpy(), y.squeeze().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3SV9Z3v8feP3ElII4RrCIRLCEMjEI2UEVQoIBatctDi4BmmdjiLth5H1CnT0p5TpzOeWVUHIopLQeuli57a8cbpsXOQcBVFKEQQpZidiwSSGJMQEnIhYWfv3/mDAAIJueyd/ey983mtxZJsnv0837WFz/qtZ3+f789YaxERkdDXz+kCRETEPxToIiJhQoEuIhImFOgiImFCgS4iEiYiA3mx5ORkm5aWFshLiogEndPNbr463UKz20NMZD+GJsbyjbioDo/Py8urttYO7uy8AQ30tLQ0Dhw4EMhLiogEBWstu1xV5OS6qC6tY+qg/qyYk85dU1OI6Geu+l5jTElXrhHQQBcR6Yv2FFazJtfFgZJTpCTF8cTd17LoupFERfj3rrcCXUSkl+w/VsPqLfnsLa5hWGIsjy/MZHF2KtGRvfP1pQJdRMQPNh0s46n38imvPUNyQgwDE6LJr6gnOSGGX94xifu+NYrYqIherUGBLiLio00Hy1j19qeccXsAqGpooaqhhTunjOCJuycTF927QX6e2hZFRHz0b/959EKYf11eyamAhTlohS4i0mNFVQ08vbWAyvqWdv+8vPZMQOtRoIuIdFPJyUbWbitg08EyYqMiSIiJpKGl9YrjRiTFBbQuBbqISBeVnmpi3fZC3sgrJbKfYdnMMfzwlnF8UFB9yT10gLioCFbOzwhofQp0EQlZX+8sGZEUx8r5GSzMSvH7dSrqmnluRyGv7z+OwbB0+mgemDWOIYmxABeuGYharkaBLiIh6fLOkrLaM6x6+1MAvwVpVX0Lz+8sYuO+Erxey+IbUnlw9vh2b6UszEoJeIBfToEuIiHpqffyr+gsOeP28NR7+T4Ha03jWda/X8Rv95Rw1uNlUVYKD81JJ3Vgf5/O29sU6CISkjrqIPGls6Suyc1LHxTz8gdf0OT2cNeUEayYO4ExyfE9PmcgdRroxpiXgTuASmttZttrTwHfBc4CRcAPrLW1vVmoiMjXjUiKo6yd8O5JZ0l9s5tXPjzGi7uLqW9u5fZrh/Pw3HTShw7wR6kB05UHi14FbrvstVwg01o7GXABq/xcl4jIVa2cn0HcZY/Sd7ezpOlsK8/vLOKmJ3ewJtfF9LGD+M+HbuK5/3pdyIU5dGGFbq193xiTdtlrW772417gHv+WJSJydb50ljS7PWzcW8ILu4qobjjLrIzBPDpvApNHJvV22b3KH/fQ/x74Q0d/aIxZDiwHGDVqlB8uJyJyTnc7S1paPfxh/wnWbS+ksr6FGeMHsX7eBK4fPbAXqwwcnwLdGPMLoBX4XUfHWGs3ABsAsrOzrS/XExHpCbfHy5t5pTy7rYDyumampQ3kmSVZTB87yOnS/KrHgW6M+T7nviydY61VUItI0Gn1eNl0qJxnthVwvKaJqalJPHHPZGaOT8aYq+8SFIp6FOjGmNuAnwK3WGub/FuSiIhvPF7Lu4fLWbu1gOLqRjJTEnn5/mxmZwwJyyA/rytti78HZgHJxphS4DHOdbXEALltH85ea+2PerFOEZFOeb2W945UkLPVheurBjKGDuCFv72e+d8cGtZBfl5XulyWtPPyb3qhFhGRHrHWsvVoJTm5Lv7y5WnGDY7n2SVZ3H7tcPp1sgGzPwRqpkxn9KSoiIQsay27XFXk5Lr4pLSO0YP6s2bxFO6amkJEAIIcAjNTpqsU6CISkvYUVbNmi4sDJadISYrjibuvZdF1I4mKCOxGbL05U6a7FOgiElL2H6thzRYXHxWfZFhiLP+6MJN7s1OJjnRmR83emCnTUwp0EQkJh07UsnpLPrsLqklOiOGXd0zivm+NIjYqcHt2tsefM2V8pUAXkaD2WVkdObkutn1eyTX9o1j1nYn83V+nBXTz5atZOT8jKHYrAgW6iASp/Ip6cnJdbD5SQWJsJD+5dQL3zxhDQkxwxVaw7FYECnQRCTJFVQ08vbWAdw+XEx8dyUNz0lk2cwzfiIu65LhgaRWE4NitCBToIhIkSk42snZbAZsOlhETGcGPbhnH8pvGck189BXHBlOrYDBRoIuIo0pPNbFueyFv5JUS2c+wbOYYfnjLOJITYjp8TzC1CgYTBbqIOKKirpnndhTy+v7jGAxLp4/mgVnjGJIY2+l7g6lVMJgo0EUkoKrqW3h+ZxEb95Xg9Vq+l53KP3x7fLfa/IKpVTCYKNBFJCBqGs+y/v0ifrunhLMeL4uyUnhoTjqpA/t3+1zB1CoYTBToItKr6prcvPRBMS9/8AVNbg93TRnBirkTGJMc3+NzBlOrYDBRoItIr6hvdvPKh8d4cXcx9c2t3H7tcB6em+63zZeDpVUwmCjQRcSvms628tqeEta/X0Rtk5t5k4byyNwJTBqR6HRpYU+BLiJ+0ez2sHFvCS/sKqK64SyzMgbz6LwJTB6Z5HRpfYYCXUR80tLq4Q/7T7BueyGV9S3MGD+I9fMmcP3ogU6X1uco0EWkR9weL2/mlfLstgLK65qZljaQZ5ZkMX3sIKdL67MU6CLSLa0eL5sOlfPMtgKO1zQxNTWJJ+6ZzMzxyX1i385gpkAXkS7xeC3vHi5n7dYCiqsbyUxJ5OX7s5mdMURBHiQU6CJyVV6vZfORCp7e6sL1VQMZQwfw9zPS2PxZBctePaAe8CCiQBeRdllr2Xq0kjW5Lo5+eZpxg+N5dkkWbo+XX7zzWbcnHQbTuNtwZay1AbtYdna2PXDgQMCuJyLdZ61ll6uKnFwXn5TWMXpQf1bMSeeuqSlE9DPM+PX2dueopCTF8eHPvt3uOS8fd3teUlwU/3znNxXsnTDG5Flrszs7Tit0EblgT2E1a3JdHCg5RUpSHE/cfS2LrhtJVMTFDZh7MumwvXG3ALVn3Jpj7kedBrox5mXgDqDSWpvZ9tpA4A9AGnAMWGytPdV7ZYpIb9p/rIbVW/LZW1zDsMRY/nVhJvdmpxId2e+KY3sy6fBqYa855v5z5f+tK70K3HbZaz8Dtllr04FtbT+LSIg5dKKWpb/Zx/de+IjCykZ+ecckdq6cxdLpo9sNczg36TAu6tINmjubdNjZWNu+PsfcXzpdoVtr3zfGpF328l3ArLbfvwbsBH7qx7pEpBd9VlZHTq6LbZ9Xck3/KFZ9ZyJ/99dpxEVHdPrenkw6bG/c7df19Tnm/tLTe+hDrbVfAlhrvzTGDPFjTSLSS/Ir6snJdbH5SAWJsZH85NYJ3D9jDAkx3YuC7k46PH/sL975lMazl4a65pj7T69/KWqMWQ4sBxg1alRvX05E2lFU1cDTWwt493A58dGRrJiTzrKbxpAYGxXQOryXNdUZ4O7rNQbXX3oa6F8ZY4a3rc6HA5UdHWit3QBsgHNtiz28noj0QMnJRtZuK2DTwTJioyL48S3jWH7zWJL6Rwe8lvY6XSyw4/OqgNcSrnoa6H8Evg/8uu2//8dvFYmIz0pPNbFueyFv5JUS2c+wbOYYfnjLOJITYhyrSRs7976utC3+nnNfgCYbY0qBxzgX5P9hjFkGHAe+15tFikjXVNQ189yOQl7ffxyDYen00TwwaxxDEmOdLk0bOwdAV7pclnTwR3P8XIuI9FBVfQvP7yxi474SvF7L4htSeXD2+KAKS23s3Pv0pKhICKtpPMv694v47Z4Sznq8LMpK4aE56aQO7A8E1/yUnrQ7BlP9oUCBLhKC6prcvLi7mFc+/IImt4e7poxgxdwJjEmOv3DM5fNTujpEqzd1p90xGOsPdgp0EYd1ZxVa3+zmlQ+P8eLuYuqbW7n92uE8PDed9KEDrji2va6SUHrMPtTrd4ICXcRBXV2FNp1t5bU9Jax/v4jaJjfzJg3lkbkTmDQiscNzh3pXSajX7wQFuoiDOluFNrs9bNxbwvM7izjZeJZZGYN5dN4EJo9M6vTcod5VEur1O6Erw7lEpJd0tNosqz3Da3uOcfOTO3j8T0eZOHwAb/34Rl79wbQuhTn0bIhWMAn1+p2gFbqIgzpahUYYw2N/PMK0tIE8sySL6WMHdfvcPekqCSahXr8TFOgiDupoCuHIa+J4/L9kMnN8crc3YA6nVr/uDgHr6xToIg767pQR5JWc4vd/Pk6r1xIVYbj/xjR+vuCvuh3koFa/vk6BLuIAr9ey+UgFObkuCiobmDhsAA/PncD8bw7tUZCfp1a/vk2BLhJA1lq2Hq1kTa6Lo1+eZtzgeNbdl8WCzOH069fzID9PrX59mwJdJACstexyVZGT6+KT0jpGD+rPmsVTuGtqChF+CPLz1OrXtynQRXrZnsJqVue6yCs5RUpSHE/cfS2LrhtJVIT/u4Y1AKtvU6CL9JL9x2pYvSWfvcU1DEuM5fGFmSzOTu1w82V/UKtf36ZAF/GzQydqWb0ln90F1SQnxPDLOyZx37dGERvV+QbM/qBWv75LgS7iJ5+V1ZGT62Lb55UMjI/m5wsmsnR6GnHRgQlyEQW6iI/yK+rJyXWx+UgFibGRrJyfwfdvTCMhRv+8JLD0N06kh4qqGnh6awHvHi4nITqSFXPSWXbTGBJjo5wuTfooBbpIN5WcbGTttgI2HSwjNiqCH98yjuU3jyWpf7TTpUkfp0AX6aLSU02s217IG3mlRPYzLJs5hh/eMo7khBinSxMBFOginaqoa+a5HYW8vv84BsPS6aN5YNY4hiTGOl2ayCUU6CIdqKxv5oWdxWzcV4LXa1l8QyoPzh6vpy4laCnQJWT11pjYjXtL+PX/+5yGllYApo0ZyOrvTSF1YH+fzy3SmxToEpJ6Y0xsXZObn7z5Cbl/+eqS1z8trSOv5JQCXYKetqCTkHS1MbHdVd/sZu3WAmY+uf2KMPflvCKB5tMK3RjzCPDfAAt8CvzAWtvsj8JErsYfY2Kbzrby6p5jbHi/mNomN/MmDW030Lt7XhGn9HiFboxJAR4Csq21mUAE8Df+Kkzkajr6YrIrX1g2uz28tLuYm57YwZOb85mamsQfH5zBi3+XTYoP5xVxmq+3XCKBOGNMJNAfKPe9JJHO9WRH+JZWD6/tOcbNT+7g8T8dZeLwAbz14xt59QfTmDwyqcfnFQkWPb7lYq0tM8b8O3AcOANssdZuufw4Y8xyYDnAqFGjeno5kUt0Z0ys2+PljQOlrNteQHldM9PSBvLMkiymjx3k03lFgo2x1vbsjcZcA7wF3AvUAm8Ab1prN3b0nuzsbHvgwIEeXU+ku1o9Xt45WMYz2ws4UXOGqalJ/OOtE5g5PtmnfTtFAs0Yk2etze7sOF++FJ0LfGGtrWq74NvAjUCHgS4SCB6v5d3D5azdWkBxdSOZKYn86v5vMjtjiIJcwpovgX4cmG6M6c+5Wy5zAC2/xTFer2XzkQpycl0UVDYwcdgA1i+9nlsnDVWQS5/gyz30fcaYN4GPgVbgILDBX4WJdJW1lq1HK1mT6+Lol6cZNziedfdlsSBzOP38uAGzSLDzqQ/dWvsY8JifahHpFmstu1xV5OS6+KS0jrRB/cm5dwp3TkkhQkEufZAe/ZeQtKewmtW5LvJKTpGSFMeTd09m0XUpREbo4WfpuxToElL2H6th9ZZ89hbXMCwxlscXZrI4O5XoSAW5iAJdQsKhE7Ws3pLP7oJqkhNieOy7k1gybRSxUdqAWeQ8BboEtc/K6sjJdbHt80oGxkfz8wUTWTo9jbhoBbnI5RToElTOzzgvqz1DbFQ/mt1evhEXxcr5GXz/xjQSYvRXVqQj+tchQWPTwTJ++tZhWlq9ADS7vUT2M/zstoks+ZbGRoh0Rt8kSVAoOdnIqrc/vRDm57V6Let2FHbrXJsOljHj19sZ87M/MePX29l0sMyfpYoELa3QxVGlp5pYt72QN/JK8XjbnyvUnVnkvbGTkUio0ApdHFFR18z/2PQps/99J29/XMbS6aMZlhjb7rHdmUXuz52MREKNVugSUJX1zTy/s4jf7TuO12tZfEMqD84ez4ikOKamJl2yuobuzyL3x05GIqFKgS4BUdN4lvW7injto2O4PZZFWSk8NCf9ko2X/TGLfERSHGXthLd2HJK+QIEuvaquyc2Lu4t55cMvaHJ7uGvKCFbMncCY5Ph2j1+YleLTve6V8zN8XuWLhCoFuvSK+mY3L39wjJc+KKa+uZXbrx3Ow3PTSR86oFevqx2HpC9ToItfNZ1t5dU9x9jwfjG1TW7mTRrKI3MnMGlEYsBq8HWVLxKqFOjiF81uDxv3lvD8ziJONp5lVsZgHp034cLmyyLS+xTo4pOWVg+v//kEz+0opLK+hZnjk3lk3gSuH32N06WJ9DkKdOkRt8fLGwdKWbe9gPK6ZqalDeSZJVlMHzvI6dJE+iwFunRLq8fLOwfLeGZ7ASdqzpA1Kokn75nCjPGDtG+niMMU6NIlHq/l3cPlrN1aQHF1I5kpifzL/ZnMyhisIBcJEgp0uSqv17L5SAU5uS4KKhuYOGwA65dez62ThirIRYKMAl3aZa1l69FK1uS6OPrlacYNjmfdfVksyBxOP23ALBKUFOhyCWstu1xV5OS6+KS0jrRB/cm5dwp3TkkhQkEuEtQU6HLBnsJqVue6yCs5RUpSHE/ePZlF16UQGaGhnCKhQIEu7D9Ww+ot+ewtrmFYYiyPL8xkcXYq0ZEKcpFQ4lOgG2OSgJeATMACf2+t/cgfhUnvO3SiltVb8tldUE1yQgyPfXcSS6aNIjZKGzCLhCJfV+hrgc3W2nuMMdFA/87eIM77rKyOnFwX2z6vZGB8ND9fMJGl09OIi1aQi4SyHge6MSYRuBm4H8BaexY465+ypDd8XnGanFwX7x35im/ERbFyfgbfvzGNhBjdeRMJB778Sx4LVAGvGGOmAHnACmtt49cPMsYsB5YDjBqlndudUFTVwNNbC3j3cDkJ0ZGsmJPOspvGkBgb5XRpIuJHxtr2N+bt9I3GZAN7gRnW2n3GmLXAaWvt/+zoPdnZ2fbAgQM9q1S6reRkI2u3FbDpYBmxURHcf2May28eS1L/aKdLE5FuMMbkWWuzOzvOlxV6KVBqrd3X9vObwM98OJ/4SempJp7dVsibH5cS2c+wbOYYfnTLOAYlxDhdmoj0oh4HurW2whhzwhiTYa3NB+YAf/FfadJdFXXNPLejkNf3H8dgWDp9NA/MGseQxFinSxORAPD127B/AH7X1uFSDPzA95Kkuyrrm3lhZzEb95Xg9VoW35DKg7PHa2NkkT7Gp0C31h4COr2vI72jpvEs63cV8dpHx3B7LIuyUnhoTjqpA9U9KtIXqV8tBNU1uXlxdzGvfPgFTW4PC6eeC/IxyfFOlyYiDlKgh4hNB8t4YvPnfFnXjOHcY7m3Tx7OI3PTGT9kgNPliUgQUKCHgD/sP84v3vmMVu+5FlMLxET2Y95fDVWYi8gFmr4UxJrdHl7aXcyqtz+9EObntbR6eeq9fIcqE5FgpBV6EGpp9fD6n0/w3I5CKutbOjyuvPZMAKsSkWCnFXoQcXu8/O99x5n91E4e++MR0pLjeX35dFI6aD9UW6KIfJ1W6EGg1ePlnYNlPLO9gBM1Z8galcST90xhxvhBGGNYOT+DVW9/yhm358J74qIiWDk/w8GqRSTYKNAd5PFa3j1cztqtBRRXN5KZksi/3J/JrIzBl2zAvDArBYCn3sunvPYMI5LiWDk/48LrIiKgQHeE12vZfKSCnFwXBZUNTBw2gPVLr+fWSUMvCfKvW5iVogAXkatSoAeQtZatRyvJyXXxly9PM25wPOvuy2JB5nD6aQNmEfGRAj0ArLXsclWRk+vik9I60gb1J+feKdw5JYUIBbmI+IkCvZftKaxmda6LvJJTpCTF8eTdk1l0XQqREWowEhH/UqD3kv3Hali9JZ+9xTUMS4zl8YWZLM5OJTpSQS4ivUOB7meHTtSyeks+uwuqSU6I4bHvTmLJtFHERmkDZhHpXQp0H2w6WHahlTA5IYYhiTEcKT/NwPhofr5gIkunpxEXrSAXkcBQoPfQpoNllzzsU9XQQlVDC7dfO5wn7plMQow+WhEJLN3Q7aF/+8+jlzy5ed6hE7UKcxFxhJKnm0pONrJ2W0GHQ7M0MEtEnKJA76LSU008u62QNz8uJbKfISEmkoaW1iuO08AsEXGKAr0TFXXNPLejkNf3H8dgWDp9NA/MGseeopMamCUiQUWB3oHK+mZe2FnMxn0leL2WxTek8uDs8RdW4BqYJSLBRoF+mZrGs6zfVcRrHx3D7bEsyjq3AXPqwP5XHKuBWSISTBTobeqa3Ly4u5hXPvyCJreHu6aMYMXcCYxJjne6NBGRLunzgV7f7OblD47x0gfF1De3cvu1w3l4bjrpQ7X5soiElj4b6E1nW3l1zzE2vF9MbZObeZOG8sjcCUwakeh0aSIiPeJzoBtjIoADQJm19g7fS+pdzW4PG/eW8PzOIk42nmVWxmAenTeBySOTnC5NRMQn/lihrwCOAkG9tG1p9fD6n0/w3I5CKutbmDk+mUfmTeD60dc4XZqIiF/4FOjGmJHA7cD/Ah71S0V+5vZ4eeNAKeu2F1Be18y0tIE8sySL6WMHOV2aiIhf+bpCfxr4J6DDbxCNMcuB5QCjRo3y8XJd1+rx8s7BMp7ZXsCJmjNMTU3iiXsmM3N8cof7doqIhLIeB7ox5g6g0lqbZ4yZ1dFx1toNwAaA7Oxs29PrdZXHa3n3cDlrtxZQXN1IZkoiv7r/m8zOGKIgF5Gw5ssKfQZwpzFmARALJBpjNlpr/9Y/pXWP12vZfKSCnFwXBZUNTBw2gPVLr+fWSUMV5CLSJ/Q40K21q4BVAG0r9J84EebWWrYerWRNroujX55m3OB41t2XxYLM4fTTBswi0oeEbB+6tZZdripycl18UlpH2qD+5Nw7hTunpBChIBeRPsgvgW6t3Qns9Me5umJPYTWrc13klZwiJSmOJ++ezKLrUoiM0H4dItJ3hdQKff+xGlZvyWdvcQ3DEmN5fGEmi7NTiY5UkIuIhESgHzpRy+ot+ewuqCY5IYZf3jGJ+741itgobcAsInJeUAf6Z2V15OS62PZ5JQPjo/n5goksnZ5GXLSCXETkckEZ6J9XnCYn18V7R74iMTaSlfMz+P6Nadp8WUTkKoIqIYuqGnh6awHvHi4nPjqSFXPSWXbTGBJjo5wuTUQk6AVFoJecbGTttgI2HSwjNiqCH98yjuU3jyWpf7TTpYmIhAxHA730VBPPbivkzY9LiexnWDZzDD+8ZRzJCTFOliUiEpIcCfSKumbW7SjgD/tPYDAsnT6aB2aNY0hirBPliIiEhYAGeqvX8qv/e4Tf7TuO12tZfEMqD84ez4ikuECWISISlgIa6PkV9fz2oxIWZaXw0Jx0Ugf2D+TlRUTCWkADPTE2kq2P3sKY5PhAXlZEpE8I6DPzqQP7K8xFRHpJULQtbjpYxlPv5VNee4YRSXGsnJ/BwqwUp8sSEQkpjgf6poNlrHr7U864PQCU1Z5h1dufAijURUS6wfExhU+9l38hzM874/bw1Hv5DlUkIhKaHA/08toz3XpdRETa53igd9SDrt50EZHucTzQV87PIO6yueZxURGsnJ/hUEUiIqHJ8S9Fz3/xqS4XERHfOB7ocC7UFeAiIr5x/JaLiIj4hwJdRCRMKNBFRMKEAl1EJEz0ONCNManGmB3GmKPGmCPGmBX+LExERLrHly6XVuAfrbUfG2MGAHnGmFxr7V/8VJuIiHRDj1fo1tovrbUft/2+HjgKqPdQRMQhfrmHboxJA7KAfe382XJjzAFjzIGqqip/XE5ERNrhc6AbYxKAt4CHrbWnL/9za+0Ga222tTZ78ODBvl5OREQ64FOgG2OiOBfmv7PWvu2fkkREpCd86XIxwG+Ao9baNf4rSUREesKXFfoMYCnwbWPMobZfC/xUl4iIdFOP2xattR8Axo+1iIiID/SkqIhImFCgi4iECQW6iEiYUKCLiIQJBbqISJhQoIuIhAkFuohImFCgi4iECQW6iEiYUKCLiIQJBbqISJhQoIuIhAkFuohImFCgi4iECQW6iEiYUKCLiIQJBbqISJhQoIuIhAkFuohImFCgi4iECQW6iEiYUKCLiIQJBbqISJhQoIuIhAkFuohImPAp0I0xtxlj8o0xhcaYn/mrKBER6b4eB7oxJgJ4DvgOMAlYYoyZ5K/CRESke3xZoU8DCq21xdbas8DrwF3+KUtERLor0of3pgAnvvZzKfCtyw8yxiwHlrf92GKM+cyHa4aTZKDa6SKChD6Li/RZXKTP4qKMrhzkS6Cbdl6zV7xg7QZgA4Ax5oC1NtuHa4YNfRYX6bO4SJ/FRfosLjLGHOjKcb7ccikFUr/280ig3IfziYiID3wJ9P1AujFmjDEmGvgb4I/+KUtERLqrx7dcrLWtxpgHgfeACOBla+2RTt62oafXC0P6LC7SZ3GRPouL9Flc1KXPwlh7xW1vEREJQXpSVEQkTCjQRUTCREACXSMCLjLGvGyMqezr/fjGmFRjzA5jzFFjzBFjzAqna3KKMSbWGPNnY8wnbZ/Fr5yuyWnGmAhjzEFjzLtO1+IkY8wxY8ynxphDXWld7PV76G0jAlzAPM61Ou4Hllhr/9KrFw5SxpibgQbgt9baTKfrcYoxZjgw3Fr7sTFmAJAHLOyLfy+MMQaIt9Y2GGOigA+AFdbavQ6X5hhjzKNANpBorb3D6XqcYow5BmRba7v0gFUgVugaEfA11tr3gRqn63CatfZLa+3Hbb+vB45y7unjPsee09D2Y1Tbrz7brWCMGQncDrzkdC2hJhCB3t6IgD75D1faZ4xJA7KAfc5W4py2WwyHgEog11rbZz8L4GngnwCv04UEAQtsMcbktY1RuapABHqXRgRI32SMSQDeAh621p52uh6nWGs91tqpnHviepoxpk/ejjPG3AFUWmvznK4lSMyw1l7Huam2/73tlm2HAhHoGhEg7Wq7X/wW8Dtr7dtO1xMMrLW1wE7gNq4PJOMAAADxSURBVIdLccoM4M62e8evA982xmx0tiTnWGvL2/5bCbzDuVvYHQpEoGtEgFyh7YvA3wBHrbVrnK7HScaYwcaYpLbfxwFzgc+drcoZ1tpV1tqR1to0zmXFdmvt3zpcliOMMfFtDQMYY+KBW4Grdsf1eqBba1uB8yMCjgL/0YURAWHLGPN74CMgwxhTaoxZ5nRNDpkBLOXcCuxQ268FThflkOHADmPMYc4tgHKttX26XU8AGAp8YIz5BPgz8Cdr7earvUGP/ouIhAk9KSoiEiYU6CIiYUKBLiISJhToIiJhQoEuIhImFOgiImFCgS4iEib+P155f9NCOxhTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.169923782348633 3.0968210697174072\n"
     ]
    }
   ],
   "source": [
    "# 随机初始化参数\n",
    "w=t.rand(1,1,requires_grad=True)\n",
    "b=t.rand(1,1,requires_grad=True)\n",
    "losses=np.zeros(500)\n",
    "\n",
    "lr=0.005\n",
    "\n",
    "for ii in range(500):\n",
    "    x,y=get_fake_data(batch_size=32)\n",
    "    \n",
    "    #forward: calculate\n",
    "    y_pred=x.mm(w)+b.expand_as(y)\n",
    "    loss=0.5*(y_pred-y)**2\n",
    "    loss=loss.sum()\n",
    "    losses[ii]=loss.item() #Returns the value of this tensor as a standard Python number. This only works\n",
    "    \n",
    "    loss.backward()\n",
    "    \n",
    "    # 更新梯度\n",
    "    w.data.sub_(lr*w.grad.data) #参数学习不需要用来记录然后求梯度，故直接修改data\n",
    "    b.data.sub_(lr*b.grad.data)\n",
    "    # 梯度清零\n",
    "    w.grad.data.zero_()\n",
    "    b.grad.data.zero_()\n",
    "    \n",
    "    if ii%50 ==0:\n",
    "        # 画图\n",
    "        display.clear_output(wait=True)\n",
    "        x = t.arange(0, 6).view(-1, 1).float()\n",
    "        y = x.mm(w.data) + b.data.expand_as(x)\n",
    "        plt.plot(x.numpy(), y.numpy()) # predicted\n",
    "        \n",
    "        x2, y2 = get_fake_data(batch_size=20) \n",
    "        plt.scatter(x2.numpy(), y2.numpy()) # true data\n",
    "        \n",
    "        plt.xlim(0,5)\n",
    "        plt.ylim(0,13)   \n",
    "        plt.show()\n",
    "        plt.pause(0.5)\n",
    "        \n",
    "print(w.item(), b.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 50)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD8CAYAAABuHP8oAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO19d5wcxZn283bPzAZJKyGxEhJCiCAQWYBIBtsgk4Px2TgA9uEz92HucDrjAOezD5wO+2xwwAmb5IzB+EwwUQhMBkmAkJCwBBJISChrV6sNE7q+P7qru6q6qqdndmZ3RlvP7yftTE+H6u6qt5566n3fIsYYLCwsLCyaD85wF8DCwsLCojpYA25hYWHRpLAG3MLCwqJJYQ24hYWFRZPCGnALCwuLJoU14BYWFhZNikyanYhoFYDtAEoAioyx2UQ0HsBtAKYDWAXgQ4yxrfUppoWFhYWFikoY+EmMsVmMsdnB9ysAzGWMzQAwN/huYWFhYTFEGIyEci6AW4PPtwJ43+CLY2FhYWGRFpQmEpOIVgLYCoAB+AVj7AYi2sYYGyfss5Uxtovm2EsAXAIAo0aNOnLmzJlVF/blt7owcUwLJnW0Vn0OCwsLi2bDggULNjHGOtXtqTRwAMczxtYS0UQADxHRsrQXZozdAOAGAJg9ezabP39+2kNj2PvKe3HZSfvi8lP3r/ocFhYWFs0GInpDtz2VhMIYWxv83QDgLwCOBrCeiCYHJ58MYENtilquLENxFQsLC4vGR1kDTkSjiGgM/wzgVACLAdwF4KJgt4sA/LVehRTKUu9LWFhYWDQN0kgokwD8JTCeGQC/Z4zdT0TPA/gTEV0M4E0AH6xfMSMwWApuYWFhAaQw4Iyx1wEcptm+GcB76lEoEyz/trCwsIjQdJGYVgO3sLCw8NFUBtxK4BYWFhYRmsqAW1hYWFhEaDoDbhUUCwsLCx9NZcDJTmNaWFhYhGgqAw7YSUwLCwsLjuYy4JaAW1hYWIRoLgMOG8hjYWFhwdFUBtwScAsLC4sITWXAAVg3FAsLC4sATWXAbSCPhYWFRYSmMuAWFhYWFhGazoBbBcXCwsLCR1MZcBvIY2FhYRGhqQw4AKRZw9PCwsJiJCC1AScil4heIKJ7gu+3ENFKInox+DerfsXkZaj3FSwsLCyaB2kXNQaAzwJYCqBD2PZFxtgdtS1SMiwBt7CwsPCRioET0VQAZwH4VX2LU6Ycw3lxCwsLiwZDWgnlBwC+BMBTtn+LiBYR0XVE1FLbollYWFhYJCHNqvRnA9jAGFug/HQlgJkAjgIwHsCXDcdfQkTziWj+xo0bB1te60ZoYWFhESANAz8ewHuJaBWAPwKYQ0S/ZYytYz4GANwM4GjdwYyxGxhjsxljszs7OwdVWLKzmBYWFhYhyhpwxtiVjLGpjLHpAD4C4BHG2EeJaDIAkG9V3wdgcV1LGpZnKK5iYWFh0fioxAtFxe+IqBP+3OKLAC6tTZHMsPzbwsLCIkJFBpwx9iiAR4PPc+pQnvJlsCq4hYWFBYBmi8S0FNzCwsIiRHMZcFgN3MLCwoKjqQy4JeAWFhYWEZrKgFtYWFhYRLAG3MLCwqJJ0VQG3AbyWFhYWERoKgMO2HzgFhYWFhxNZcAtAbewsLCI0FQGHLDJrCwsLCw4msqAWwJuYWFhEaGpDDhgA3ksLCwsOJrKgFsvFAsLC4sITWXALSwsLCwiNJ0Bt9kILSwsLHw0lQG3AoqFhYVFhKYy4ICdxLSwsLDgSG3AicgloheI6J7g+15E9CwRLSei24goV79i8jLU+woWFhYWzYNKGPhnASwVvn8HwHWMsRkAtgK4uJYFM8EScAsLCwsfqQw4EU0FcBaAXwXfCcAcAHcEu9wKf2HjOsNScAsLCwuOtAz8BwC+BMALvk8AsI0xVgy+rwGwu+5AIrqEiOYT0fyNGzcOqrAWFhYWFhHKGnAiOhvABsbYAnGzZletusEYu4ExNpsxNruzs7PKYornG/QpLCwsLHYKpFmV/ngA7yWiMwG0AuiAz8jHEVEmYOFTAaytXzF92ElMCwsLiwhlGThj7ErG2FTG2HQAHwHwCGPsQgDzAJwX7HYRgL/WrZRyiYbmMhYWFhYNjsH4gX8ZwOeJaAV8TfzG2hTJDEvALSwsLCKkkVBCMMYeBfBo8Pl1AEfXvkjlyjDUV7SwsLBoTDRVJKbVwC0sLCwiNJUBBywDt7CwsOBoKgNOVgW3sLCwCNFUBtzCwsLCIkLTGXCbD7zx0TNQxMbtA8NdDAuLnR5NZcDtJGZz4LqH/oF/vum54S6GhcVOj6Yy4ICdxGwGbO3NY+uO/HAXw8Jip0dTGXBLwJsDjAEl29NaWNQdTWXAARtI3wxgjKHk2TdlYVFvNJUBJyuCNwU8BmvALSyGAE1lwC2aAwyAZw24hUXd0XQG3EqrjQ+PMauBW1gMAZrOgFs0ARhQtAzcwqLuaDoDbgN5Gh8eY1ZCsbAYAjSVAbdzmM0B60ZoYTE0SLMmZisRPUdELxHREiK6Oth+CxGtJKIXg3+z6l9cWD/CJoDHGBizE5kWFvVGmgUdBgDMYYz1EFEWwBNEdF/w2xcZY3fUr3gyLANvDnCzXWIMjg2/srCoG9KsickYYz3B12zwb9ioleV0jQ8WyCfWF9zCor5IpYETkUtELwLYAOAhxtizwU/fIqJFRHQdEbUYjr2EiOYT0fyNGzcOqrA2H3hzgMvfntXBLSzqilQGnDFWYozNAjAVwNFEdDCAKwHMBHAUgPHwFznWHXsDY2w2Y2x2Z2dnjYpt0cjwLAO3sBgSVOSFwhjbBn9R49MZY+sCeWUAwM0YogWOmWV1DY9QA7cGfETgjc07MP2Ke/HM65uHuygjDmm8UDqJaFzwuQ3AyQCWEdHkYBsBeB+AxfUsqH+tel/Bohbgdtsa8JGBp1/zDfedC9cMc0lGHtJ4oUwGcCsRufAN/p8YY/cQ0SNE1Ak/y+uLAC6tYzlDWJPQ+AgnMe1oaUSAv2U7RzX0KGvAGWOLAByu2T6nLiVKgK0ezYFwEtMb3nJYDA34+7Yj5KFHU0ViAjaZVTOAT2IWrQW3sKgrmsqA23zgzQHLwEcWeH4i2zyHHk1lwAGrgTcDPKuBjyhEr9la8KFGUxnwWlSP51dtwf2L367BmSxMsG6EIwvhJKa130OONF4oOxU++POnAQCrrjlrmEuy88KG0o8wBO/b2u+hR1MxcCA5kOfU6x7Dp//wwhCWZuTgst8vxF5X3ptqX2b9wEcULAMfPjSXAS9TQf6xvgd3v7R2aMoywnDvonWpPYC4Bm5zoYwM2Nc8fGguAw47idkM4O/ILqs2MmDf8/ChqQy4HaE1B0ZKKP1Vdy3BJ255friLMewolHx/URuJOfRoKgMOwFLwYUCxVKFD9wiRUG55ahUeWbZhuIsxLFjf3Y9Trn0Ma7b2olAMDLi130OOpjLgNpBneLB5R76i/UcKAx/J+PPCNVi+oQe/efqNkIHv5P11Q6KpDDhHX76EXz3+ul1zcYiwcftARfvzyDxrwIcHjLG6p11uz7oAgL5CCfmSTZ0wXGg6A87A8J37l+Gb9y7F/UuqD8ixecXTI1+hhMLbsTXgw4Pzf/kM/veBV+t6jbZcYMDzpZCBF0r2fQ81msqAcwFlW68/pB8olsLfKjXItrKlR6XPVlzU2GLo8da2Pqze2lfXa7QKDJzPkVQ8V2IxaDSVAQd8nY3bXkfQxEW2d+Wdi8Ik8ybY4V56VD6HGUxiWgY+LPC8+j97bsD7C14ooRTs+x5ypFmRp5WIniOil4hoCRFdHWzfi4ieJaLlRHQbEeXqXVhur3nlfPCV9Tjjh4/D85jEqP/w3Gqc/8tnEs9lGXh6VOpNwne3/sHDA4+xustXWddvjH2FYiihWAY+9EjDwAcAzGGMHQZgFoDTiehYAN8BcB1jbAaArQAurl8xIzAWGZR7F63D0nXdGCh6KFTIqAu2sqVGpQbcGwEMvJHnUDzG6i5f8eYmauBFS4qGHGUNeLBwcU/wNRv8YwDmALgj2H4r/HUx6woeKKBjF9wXNS1sZUuPSm3BSNDAByqsb0MJj9V/Apl30gvf3Ia/vuinr1AlFMYYvvfAq3hrW+31eEvAfKTSwInIJaIXAWwA8BCA1wBsY4wVg13WANjdcOwlRDSfiOZv3Lhx0AVmYFDrpsdYxcN1WwHSo1oGvjN7ofTlS+V3GiawIZBQdHVClVCWrtuO6+etwGW/W1jTa7+1rQ8zvnIfbnv+zZqetxmRyoAzxkqMsVkApgI4GsABut0Mx97AGJvNGJvd2dlZfUkRaeDq8LXEGPIVMiJrwNOjYlswAgJ5+gqNa8A9Vv8oWN2rVUe1vAy1bmuvb/QFgbtfWlfT8zYjKvJCYYxtA/AogGMBjCMink98KoAhSwOoDs2ZV/mEWT0n2PoLJfQMFMvv2CQYTgbe1VvA9CvuxV9eWDPoc9USvQ3MwIdiElNXJ9R5qJ1YQWsYpPFC6SSiccHnNgAnA1gKYB6A84LdLgLw13oVUgTT6Hslxiru5evJwOd871Ec/N8P1O38Q41q/cBrwQJXbd4BALj5yVWDPlct0d/IDNyrXFKs+BopGDhHLTNg9OVLtmMQkIaBTwYwj4gWAXgewEOMsXsAfBnA54loBYAJAG6sXzFlqC+w5FVuwOs5ibm2q79u5x4M+gslfPf+ZRXrt5W6zEcMvLLjktBojVZk4I3mkcJY/T2AdPestkFW48xzm3sGcMDX7sdPH11R0/M2M8ouqcYYWwTgcM321+Hr4UMKhjgDZ4xV7Nc9EjXwXz+9Cj999DW0ZFx89uQZqY8TmTRjrGxSsWhFnsE/43Deo8HSUIoauMcAt4HyrA2FG6HWE8zQpmqVZvbXT78BAJi/amtNzrczoKkiMbnhUIfmJcYqDiIYiYE8fKI3X6qQgUsGvPz+tVxSrVFzTItuq40W1VsaEg08+jymJYMT9++MyTa17kO2BFkx99p1VG1P3MRoKgMO+JVCJ6FUmnCp0RpdI0Nsl2l07XBR4xo24AZTKSRj1WhVaSj9wAFg8rhWTBjVEpMl+bdaaeAjwT21UjSVAef1IOaFwirXtEeihFItJAaeav/gby0YeGMScMmINFrA0lD4gYsauMf80HqzhFIb8FuyKRoiNJUB54h5oWgmMcs1/JEooVSLihl4YOZr2dAazEZKI7hSg9WlofYD9zyGjEsaCaW2ZWCWgcfQhAY8nqy+VMUkpg2lTw9WoQYeMvBGs7o1RCMz8GoikyuFev8Okdlg12gYxetTNfLn9Y8sxzOvJ2cobUY0lQEPsxGqofRVuBFaCSU9hnMSMzxnzc5UG4gGspEYob8az9C6ERZLvgFXLzmYEix7uxtX370kJtUA1T3vnz/2Ou57eeeL3GwqAw7oA3k8Fu+Vy/X5I9mAV0oYxUebxp2vlkPdRmXxpYY14P7fumcjVGQ1ovi74l+r4d8f/dWzuPnJVdjUE63Hys9fadoMwLcPlTo6NAOayoBHDFyjgRcrlFAaqNE1OjwNC0oC36UWhq2RjKOIYo0klLU1ztQXpfKt6WmN1wH8Z0EgIzGoRkHh7108lp+/GkNcLLGGziBZLZrKgAO+cVANuMdYLA9DuWCTkczAK21Q4uNOMzEVunvVgAVyO9lo0Y4lof5UK1c8uWIT3nHNI/hbDYf2kadGfeu3OonpkO4dVf/OdHJJtQycBXMCO6PjQlMZcFM+cI+xivOB72wv85nXN+O4/5mLHSmSaFUsoVTKwGvoRtioEkotNPAnVmwCAKzY0FNmz/SoRxoD7XXUSUxHo4EPQkLRGetocjza7yt/eRkX3/J8mXMhOFfj5q+pFmVD6RsNfJJGRKmK5D0722ox19y3DOu6+rHs7e04cs9danpuiUmn8kKpoQbeoO9JvLdq5bj1Qc6c3Tpaa1ImQJBQ6q6BCwa8pNfAB/Pq+KmKGgbOQQT87tnyOcH5aLsa7bzR0VwM3KCBL9/Qg2de3yLvqzlemjlvUMMwFKhUQqnUDxyaxlcteOfRCES8UPLwZMCakwxLWqzf7hvw9hZ38IULy+L/HcpQ+nNmTQncCNV9uI5dOQfX5RKvVkbjz2JnG3UDTWbAOVRt9Ut3LMLDS9eXPU48rBaJlkYKJD/wFPvXkgXy19QIyax+8PA/cOGvnsXzq7bUxAvl7YCB19LYDlW4Ob/Os//5Hnz9vQeBoGPggQEfxPklCaXKJstjPurJwBet2TYsE+5NZ8AZqn+RpZ2YgfO7SUN2Kncj1LPNksdw1V1LsHpLr7x/DVlgpZ1Ad38BS9d1I1/08P6fPolnaxi8wbXqzT0DNdHAN2wfAFBbuYMFbWOoQunHj8oh4zo+A4/tM5jz+3/F51xtJ84ndAfqNDHw8pouvPf6J/HDucvrcv4kNJUB57YpTYXXGbJG9d1tdIiPSnz0L63ZhlueWoXP/+lFaX/e0GphmCr1ZPnYjc/hjB8+jtVbe7HwzW248s6XB10GDtMITqxLb27uxfQr7sVTr20qez7ODGsZFVxLD6AkcFvoBA3N0Wrg+jKs3tKL+au2aH/j4IeKEkq1TTaUUOrEwNd1+a6gP5q7PJTYhgppVuTZg4jmEdFSIlpCRJ8Ntl9FRG8R0YvBvzPrX1x9IE8lx3LsbAy8ElSugQssSPNZF1gF1MYw8WuktUcvrd4mHz/oEujORUY/8GdW+oz/zoVvpT5vLRl45Ac+NBKKE9QlCjRwXeSkWt/e+d15OO/nT6c6f6Hk4eYnV2Llph1Va+CFoCD1CuQRS3Xhr56tyzVMSOOFUgRwOWNsIRGNAbCAiB4KfruOMfa9+hVPgSEfeFqUlOH/zog0trnSx1ehE0q0qHEtGHgDTVUwwSCJnZNUlyq4ZW7YeGfQM1DE5p4B7Dmh+nzXQ5WxjwXRlxQycAq2C84GPBinChWct/Ge/iKuvvsV/GTeazh06tiqylqqswY+nDEKZRk4Y2wdY2xh8Hk7/PUwd693wYzlQbrKqas0kuuXgR3+ecEa3LGgsRbQFbFkbRc2bB/aJdtkP/Dyz76WLJCf67WNPTjoa/fH9HbjcXU0YITaynG8rBf88hm8+38fHdS5JAZcx2fgschoA3oPsfBzFbOYoe920INv7y9UTdy4Bl4vAz6cXLAiDZyIpsNfXo2PEz5FRIuI6CYi0jofE9ElRDSfiOZv3LhxUIUN84FXOTQXK3d3fwFX3bUE2/sL0j63Pb8atz3/JlZv6cUHfvYUunoL6mmGDdt68zjrR0/g6G/NjTfOQbhslYNJAzeB71ILaZffp8eAHfkS7lmULmqxPhGc4iT44CMx+ZvihGTRmq6qSxaWRdTp68gMPcZC+QSIpBQm7TP46wwUBq+B8+dbr+jr4XRxTW3AiWg0gD8D+BxjrBvAzwDsA2AWgHUAvq87jjF2A2NsNmNsdmdn56ALzMNiq4HIlO5YsAa3PLUKtz2/WtonX/JQ8hh+NHc5FryxFfcvaZwMZjuEhXSHMoWpiYGbihC5slXeYD5247O4f3H0zNX7TLt0Hm+sNTXfoYRCNU0nq1vjtepzDZFMWGLy2qi65Q7TMOY3Nu/AA0veNv4u5i+p9rnU241wOF1cUxlwIsrCN96/Y4zdCQCMsfWMsRJjzAPwSwzBAse8vqSqmDovFE0F6GjLSt8LJQ8lJrjlpRj/rdiwfUiYesmkuyIqb1Ilr5adiwxTNxGsnpfvU6kBYYzh8eWbcOlvF0bXVk5RSHnOerAt8RmLEpyOUOiedLHkaeuJboGSaiGFuNfRgDMGuMJ7FzXwaB+ugZvxnu8/hk/+ZoHx94GiuHj04CSUerkRNrSEQn7rvBHAUsbYtcL2ycJu/wRgce2Lp0e1iXp0799RjE+h5MHzhHD9FDbv5Gv/jnN/8kRwjfqyHg4z+zUfX23ZTBKKbh6BSQywsuvoIuVUeSItA68H2+L35jF5yTKpg0tgY//1f4tx2NcfDMvGOz61AxjMBKTk6lhPCcWTJRSdBs5flYk3eClSYPQX+LOKx3+kvT1RQqlH+xzOScw0XijHA/gYgJeJiDv8/ieA84loFnxisgrAJ+tSQgWMDd4fVERfXk7+VCgxZJxov7ScddXmXuM1agXJ99hQadJUpkG5EQoGSs0A6V9ff1wacNacESyDeo60xo1PftWjbRVKrKpAnv970XctLHoecgJ3UjupQslDa7a68HpJwhjCScxQA9e8f9MoNk0H018YPAPn74cxv/5k3drOEw2nBl7WgDPGnoDejv2t9sVJBiE9+9YVWNfQevNyhrJ80UPOdUJDVansUAvWs6lnAOde/yRu+ZejMGPSmHB7ktHgl60kW6AJr6ztxu67tGFsIC/pfHsBPQNX80RXAs5MXcGAq/eZVhqpVkLZuiOP3kIJu49ri/3GS1LyWMwLZXt/AX35UqLkZnruMQY+iNlfNVK2XuCLOHA4VWjgae5T1sArLGQAsS7kix6ybm3jFxteA28kDKZy6yqAasB9DTwSwSvtq2vRaB5+ZT3e2taHXz7+urRdvHcT0066fjTR5BsqHbp6CzjzR4/jsKsflDxAdNflcob4jMSrV8oAdQxcvc207z9f4QIfHO/633k4/ppHEvfpzZewvrs/ZJ0lxnDKtX/H0d+em9iBRxq6/9eUM0Y3skkLyQulzgbcEd6TWLc4yhlckyePWMfSauBJI0/xOdRjbmQ40yo1lQEnotSTWDroGldfQWHgXAOv9hqGilgJeMNQ61oa3+Oka/Lffv7Yazj8Gw9hY5CLQ0SPIClxQ2LKB66LbBsMA+TnywgMKeaFkrK1RF4olZVhe785nzovyn/+5WXMf2Mrchm/nCWP4e3ufum6+uP9E/B74s+nlgxcmoOosxuhq5VQ4gzcNIg1peQVWXe/5EaojDqhr5cqxHPXY25kOHPWN5UBB9JPYpXLhcLRq2rgRZ+BszKVzwTZgJfff+P2AbylLKvlaoajQPIyXlH+EfO11PJs2ZFHb76Iu15aG96v+Hy5rZTPKTLwyCA9sOTtWK72Sg0Ib1yiRhmXUNKdszBIDVw3elC38F1khle+3qgJp+ITtekKva6rD1++Y5F20QO1XLWGx2R5kYTt6vVNz6Jg8KoSiYHMwOXj0wZTic+zHsuqDaME3lwGXI2AqxQ6dhqXUHwXMb5nOQOunlO3BFQSjvrWw7Ehu+Poj0/TOSQZTfXRuQ7wk3kr8Jk/vIBHlm0AoDSq4FxGDTyw8C+8uQ2f/M0C3LNonZLwqVIJhQXlEiUU1bjVVwPn6MnHmbipLHKEbxID9/+qCadUBp5WQvnPO1/GbfNX44kVUYCcPIlZ/hwlj+H+xesqHi0ypgTyOBRuj/bx/xonMSUGHhXWZHDVMor3l9TWxMn/uriXWgaeHmknxrSh9DoJRTDgjDFfQhGYpHqe9d39OOSqB7Ds7W4AyayAAVjwxlbcubCy0Hw+IZTkH2wyjokVWfmNiOAGvcXzq7YCkBtSyBAN7osqG17f3S8Nays34FwDd4JrMazc1Kvsk+6cgx0qp/Hr9xSDDJSRUMLjmDRaUZ9TWgbO2wJ/h+q50shNv3l6FS797UL8uYLkW4BvPB1tII+wTxnDJj4rU7kHCgkMPOWEuVhn6pHQajgT4zWfAR/EC9DVZ5GB8xdR8swMfN6yDdjeX8RNT6wM9xUhVqofPrwcH/jZU/j8n16qqJycgSYxcKMBT6hMKlNwiNA5pgWAn2MFAArC5J9uEjPJWKkunpVqg5xtZQIJ5ddPv4Gbnlwp7ZNkIP/yQtRR5gcZx9/Vlz4wSywSNxa6WxeZd9K7TMsSuaHPGiZ90zx/npN8nSLjlUNJZeBaDTx+nBwnoDfA4vYBSR6qvD2ov9VDAx/OxHhNZcCJBtfb6Sq0yMB5w+EMSQfun8snV2I6tWA4rp+3oqpyhhq45/vBfvX/FmNdV5/ETExtM+nxqGV1KOoQ+eowBY2vuUm6UZkiAzM20DRQvVCe1+SMTnr//3Fb1FFWq4G3BBOT2wQG3tVbQHd/wSxbicN/Ly6rqGBMCXlXTrxhe3+qhGX8Wq7Bbz5NP8AnjCteU1bxQnESGDgRMH/VFty7aJ3Ehk1GW6xX3A+cEF80Oa3Pey0mMXvzRaNNGM6l2prKgAPph5dpJzGfW7UlHC5z9im+cNUPvDXrPzJesZJYQbUIvVAYw41PrMRvnnkDv3/2zcT8G6q+qkO8AUTPM0xtqplY0nkWAHGt1lMY+JYdeXT3p2ey3Ogm+emmZafVNtQxrb7v+7a+yM3ysK8/6LtVpnDd5I1Z5y0lviPpXSp1+hO3zMfR35pbtqz8WqLXjuyHX/4ZcPZeaXQzY4qEorm+WG/O+/nTuOz3CyUJQ9K9UzDwpPmmpA5IHLWnlVC29xdwzLcfxvOrtmBbbx4Hfu0B/PgRPSFLyvnDGIslzKslmsqAE6jqMHrAbNx++pj/YgZKvlGWJBRl35CBF/VMqxauWxEDZ3joFX+tzz12aZcq6ePLN2oZQRITUX/zAs0fiLT+okaXNJ0yxsAZpCn5dV39mP2Nh43lUaEG8ugum7YDr3ayqqPVj23bpmjgjJk7537J7c2vQ0lSn6eci9cZRz/XZwQ/h8zAhet4vgG5f/HbxvKEDLxCFhnPRhh/Z7rnJXasElkwMPOBBDfCNA4Dt89fjT8Lc1BpO/YtO/JY3z2A5et7Qnfbu15aq903qfP45r1LcchVD0oRpbVEUxlwUHXMqr9QQl++FHvJ0ye0A4gqH2c0nseMvkE5V2HgQnFOve6xqvyzTSgxFi7XxCCztq/9dYlUoSJ2Zz5fbMkrj8UabkHDhEzRdapR8BiLXaOSSaNQQklg4Gk78GoZOO+gdwzEvVBMDVVkWFySS+pAPI9J9SZyt9Nb8CVru7T3o7uGWM+eeX0z9rryb7j0twtw69NvaM/NXTYrlQE8lYErizjwffzfov0kA26YuBTreW8heg9qE0oTc/DFOxaFE/Tq9ZPAzzdQLIX3Yepgk9Jb3xjMlfXlrQFH1iWJ7SRBfNbHfHsuDrnqgZgm+G8n7gOiSHPla+aVGAu9KdRXw7/z2XGRcf9jfY+xIlViyMSVxcVgD9WAvLE5vrhBJW6EHhM02+C4gvUetHsAACAASURBVGbSSDxOPL06UcjY4IKKw0nMBCqa2gsl1MB1UgbDpp54EBMQuXDqnqOJxYrBPzwwLKmcHmNaDwrdbe8YKOKsHz2BD98QX4KM142SxmgCwLf+tjT83KvpkICIvd/78lpsMUTnau/B04fSl5tEFf26TeuKisZcDDardhJThOm9/On51eFiIRu3D+Dr97wCwJ/ripaP09dLU3BhoaTvrGqJpjLgOdepill19RVQ9OLsMOs6cIXczuEkphdVxK6+Aha8EfXgfN++CjXwShiOGGLNjyt5TJv0SEUS09eVlRs6bpx0TMikgasGzfdCqb6ihpqubnUAfs0KIzF1uPGJlZj9zYexatOO2G9cStKxKpNnS7fgsRJKKAnlVCWUcOkxxUB4HkNPYHhfeHObZPz8ayS/HxHtLfq0R3y0s757AJf9bqF2H/09MEm60WUjjFxxI5gklKL0PPy/uYwjRWLGGXj0Oa0Bz5fiTLhQ8vClPy/Ch3/hd5JX3vkyHn3V9633GXjyuU0auOjhVq9ozaYy4C2Z6jK0cagGMOs6cBwK2VBeMGL8eX/1/xbjAz97KmyYfF/OFtVzPvYP/apDlXQ83PZ4nhwsovbiIqvnvyROYmo0cN6ICspf8VxyNsIIank8NrjMbGkmMVNr4Am5UHjjfDNheTZdEJOJgXdXwcDVycbTf/D3WB3pDaS/8NyavD1AOi24x5AiQGT9lSzV52vgOi+U5LIMGCSUZ17fHK7ozju/KWNbY9cUUc2iGrp2yLdtDeY9xIn3/kJkC0wSl4ldp51kHQyayoDz3BNpQMJEIIf6krOug4xD4T6hBs7iOTR4xeMNWjXoHP/7wKva8lRiwEV/YdE3Xe3pdUYqSalR6xBjceYtM/D4caJBU1lukvtlGuiyEarYoYmQ1J4rYUUeMhN8Sb4CzAZHKpMgT3AjmxyRqaajBZa9vT22X09/Ef0C61ZlON0ks/r4uRE0eQOJ7WOUgaVrj2NKKH3w8cFg0h2I2oZo9/IGWeHqu18JV3Tn9zV5rJwRUn381UgoulGUKt2JE47iqKdSDbwWy+6Vw05rwDm29ka6nvoMcxmCS6RdM8+USIpv5w01dcUpY8A9DYsSE97rNHCdTJDsRqgwGMbCCh0y8WK8UZgiMWMToCUvNIpiZV+9pTeWc0aHiIFzj4b4vWzoHkjVSVQbcafqyiLrVT0Jvnz6TDgkG/m+YMifxMBLnkIsDEPwnoGiEqcgn1Mnoaj18V/fuTcmdbQYXdnEOtVWQQ7yWCh9YKWvuW8ZXly9LdjH/020e70Deg1cV6bJ4yIGTuTXwzGtUSeTl9pr9e2QG2keQNYrvfNIAzel1WhoBk5EexDRPCJaSkRLiOizwfbxRPQQES0P/moXNa4lRAOeS5nTd1NPZMBVVsQllJCBF8UKIe/Lj+Uvsz9k5OnKrtPepPMbvD9Eg5ImYk/t6X/599fxxPJNUtnDfRkL74ufS/Tt5vvLE1Pm6+eLUWUXWfQ7vzsP/3Lz87GyFkoebn5yZXieMBuhY363A0VPkixMKKQY8WjdMIXRDyBnqxS17s+dPAP/duI+yDiOkj0vhReK6gduqEM9A0VJA1brb1g3EmQL1yF0tGaNWRbFcrTn0hvwkqeXUAB/8W1AzzrFEZSpk+NlmiIwcBbIcxNG5YTja2TAC7L3k9hpDhRLwiSzSUIxdERSTEX17s9JSGMFiwAuZ4wdAOBYAJcR0YEArgAwlzE2A8Dc4HtdIRrtljJsnD9q0dtgq+Lbm3UduIIGPmAY3gG+4XjsHxvDmXpeEdJWnHJZ0HSeBGIF7+or4HolkECfzlX+/q2/LcVHb/SHpmodUhk+oA/kMQVnqLPv+VKkF6q28dmV8ajKW55chavvfgW/e8Z3cUsjoQDAxhRabZIBTVqkg98SHxaLbExcVJq7GzqOPMyOJjGTR0Ly4sMGBq5IKOo9RZHD0Tb1uTsOYUxrxiihiOU0TXTq4DEo+cDF32RpR3zeotykazuivDRWWa/WYwwTRreE32XCla4d6upFooRS8Mq6eZquLUs8qYpXMcoacMbYOsbYwuDzdgBLAewO4FwAtwa73QrgffUpYgTRaOvkFN3z3Sy4Rm0OjDk3EFnXgUMUPtyCYYacn+eim57DlXe+LG1PO3lSbr1HnV4myg43/P31MG+F7pzieo0mxBm4MHEbTmIKjULRg/kxYZkTGHia57IlYGq9BtZqOsWGbr0LoIiknCTh+TXb+LMPGbjBf7c1qH8ukRRs0pcikIcxVULRF7JnoIh+4frqIhX8nenkNw6XCGNSMvBRAQN/9NUN2lzxIuKBPOI59WUB5FGMyZjyDo1HPQORhDJeZOAVTGJ+Zs6+cB3Crx5/Hft95T7tohGuQQPn5RTvcVtvPpSlTBPrJj/3WqIiUZmIpgM4HMCzACYxxtYBvpEHMNFwzCVENJ+I5m/cqPfQSItyBlwnq3QJGjg35vw8OdeB6yA2iQnEK8S2Xr2PbNrJCW0ghiGFZjnjER6vOafHGLb15nHD31+TKqJvXOP7FhXpRPJC8eJGUFoTU6m4+aKZgetIdbEkM59oEjn5ma5PwcCfCDwadDp6WBTNZdROy6TdcwbuOiQx8CiQx3wPJU9l4Pp9t/cXEhm4PtBKPofrAB1tsgHfsL0fX7rjJfQXSlK9a89l4HkMH7/5ea3fuQjG9NkI/XJ5UmculluUv3T33d1XCMvUomjyHgN2aY9YeRoJJesSLn33Pvj8qfsj6xK6+4vIlzztAhLc+0mUzfpFBi6cd9bXHwrTQKfRwOu1ak9qA05EowH8GcDnGGPdaY9jjN3AGJvNGJvd2dlZTRlD5MoZcHFb8LTFkGgup/DGl80QMo6jncRUGdSOAb0xTTs3oY+kiw5e9FaXcE5/+45yBtyggT+8dAO+/bdl+NwfXwy3r9y0IzESk7HAVVEbSi8y/eh4lVUMCBKKCp0LqJoONV9mXoG/33Vdlbi7Jf0W/1GdxOw1hEBP6vAn2FxHZuDcGJTTwHVSlYpVm3egLx+dJyahhCsmyecW4ToORrdkpEnM7z3wKv40fw3uWbROkm9aMk54/Osb4z7y6j3oJjEB4Iu3L8J+/3VfNF8kPEMxy6PO8HX3F8PnocqkjDHJxVS8VZMcUyixsN6IBE/Kt1KQpTvxVP2FSANXJZTu/iI+fvNzRgMu53oZRgZORFn4xvt3jLE7g83riWhy8PtkABvqUkIBooHWzZjrGPg2ocJsDiY0+X7+JGZU6ZNmtXsG9BrioAIIBKN+0U3PhWyPM99yDFyngYvE79mVm8PPT6zYpJVQxGFooeRph6Wqb++8VzegUPJi5csXPeMSZi3Z+Lvhz44zcM5kVVc+jo7WDMaPyoURc2mgez+8HSbJWqHxMbyDPcb7E2yuQ9LcCUeyBi4/U1MdWrpuuzyKMkhMcrZI1YD7hlAkELzD7BOMEz9PWkmw5DHJoInGfHugc4ceW4aJYJ300N1fCMvUKrTx3nwJm3ryxonEtdv6cMYPHw9TTwDAnUGO81zgXZITSITYGYZeKJph4kDRC8upG0U++upGKWe5iDT++YNFGi8UAnAjgKWMsWuFn+4CcFHw+SIAf6198WSIBrpNM2Oue7ciA9+8YwA51wlfRM4QiQnEG6BOQ/S8eHSnCeUkFCBKUcvLU84VTndOJnmWRGV7ZNn6uB8tY1In8v9+PR93zF8dftctqbbwjW34l5ufx3fuWxZGCYrlMdkt/u6WrO3C/v91X5AelzNw/4Xw4TU/h/oOHCJMG9+uTSFggk6L5tVEx4rCVXI0k5gidh/XHpY9bZ4SDiZ4oThk1m+XruuWjJ9Ja03Kwe0QIeP4rrL3L34bc5euD5ntQKEkGZlfPbESf3j2TWO55WtCWhNT1/b4MxA7epGBlzwvdtx2gYG3akiaaX77j8+txtJ13fhtMCHOGMPlt/vphTlrzwlL9elW/cm48ZP3F0rhKIVAeGtbX6yTXNulz6UuaeB1SjmbhoEfD+BjAOYQ0YvBvzMBXAPgFCJaDuCU4HtdIfagOpenz5+yf2xbl8LA23JuWMldh6RIzKRZ7W5Ngn++ek8aXD9vRezFqwwwSlGb6pSSkeCn9sPvZbc8wJcdtJGYghF7fPkmrBXkCV00Ijcoi9Z0aQ24yUebM/DVW/owUPSwZmtfaFz5MfwZ8++qEXQdwp4TKjPgegbOMy9qGDhntZpRmQhOIFwDI+wZKOKeRfrsdSWh489lHG1A1u7j2rCuq1/S4E2dQjk3wmzGQW++hEt/uwAX3zo/fBcDRS/WSV519yvaa6jw84FH33UeGrw+i66Q3f2FMONjocRijHr1lt6IgWsdFQg3f/woHDC5Q9rO3wePOBXbFncPFEfwInmKGHj8egNFLzzXc6u24PhrHsEDS9ZL+7yxSV8f08hkg0UaL5QnGGPEGDuUMTYr+Pc3xthmxth7GGMzgr9xP7Eao0WSUGSXp8tP2Q8XHDNNKLf/t6svj8lBNNrmnjzasq7kq+xSPBITiLM/ne+xP2Oe7sUsfqsbK5XcG+okpCm/igk6CaDkRdtF9jlQiHc2XCM0QbciDx8EdfUV0NNflFz+/A5Nfy6ugYsBMvwZcwbEDXgoaWmY7Z7j27Guqy91ZGvyUltmP3oxiVgSHAMlZAz41O9fCBelEDVoj0XyQtZ1pIlKDh6w0t1X3oCLRVTro0skrdgDRME0fXmfgY9ty0qTg2lgCqUXwQ23LKEU0RG4B5Y8FkvX/ONHloeMV53E5Nc5aeZEnHyA7DPBPVZ6gnsTO95IQonsBzeuqzbtwJotPoPOGhm4/ExfWrNN+r5dITI/mrscb22TF2CpRZppHdI7fjYAxBegMnC1IXEWt623gL07R2FdVz/yJQ9tOTeUQwi+EeeNNEkD1zHwNIluRKj+zWqDrDS60xSJWVQmtzIOoa8QL6tv7M2GUDeJySfVuvoK6BkoYreOVrwVLMflG1V92bmEwsvWm48axjfvXYpj9poQjpb4GXReFxNGt8BjPsMdn8mhHLQMPPirM86qF0pJKcN/nXUAPnzUHuH3cj7rfLGQo74V5UUXfZ1bMo52roMbcHEEaUqmpVtIWCyfmluGz41s7c2H2TjLLd6twmMwauAcvGNSJzF9/+4+7fNf3z0Q3merZt6EX1LtMDhB4H7mIjnKCnNeHNyAn/i9R8NtrkOxEaTPwOU6UC5v+7UP/QNL1nbho8fuGV1vuBh4IyHRgKuZ3ILnta2vgGnj20P/0basi/95/yHYu3MUdhmVgytEYoqsTn1pd74QX/Q1X/QqctCPB2IMTkLpL5R8V7NCKUp/q2HVY1oz6C+UYudVvSFURJOYkaHibGrzjgEMFL1wdAMka+B82M6vp7qwnXP9EzENXL2PksfCyWvVvc/kzln0zPlZBgqlWICL6oWiNrxx7blw1R6gvAHnz7BfWpggml/IuY7EUDlGB0E1YvlM0aVJk2WuQ7H86qsDxrmtt4CS52cVNE0Ocix8c6vUPlQvFN0i4nxyT2Lg/QV0tPn3Vix5CouXj2vVeC7xfUxkiEd6imSsnITCkXWdWMCdjoHr7lXFK+u6lQyL1oBLL0Cd4FDbEa/I3X0FjGnNhos3tOVcnHLgJDxy+YmxSMxKQ3MHiuk1cCAeiBFj4BVKKOu6+nHIVQ/iPd9/LNymY9UdbVnfgKsauMe0FVn8HfA7Ba71RqHi/m+Tx0XhzjqWz8HlLzEdr5pbhEe5mjTwosdCrVM9NonhxBpgUFeuuvsVHHrVg/IkoGDAt/Xm8ZriTqcOs00auHo+qTxCJGYu40huiByjg06iq68QMlHxefRJqUqh/Qz4I1O1zLyebe3No1hiyDik7Yjue3kdGGNYvn473v/Tp/BtIb+4J9QJwMDAC3G30O39xbADLHhMcq7mbZobUf0kpn+AWl5+DJ+XyUsMPArc49ARF9UllJ9XjThOs0Ta6i19kneKZeAAWlwzA1dfKGN+Dz9Q9DAql8H0XUcBiLsfOgYvlDRLci1fv72intUUCs1hklB02hwQVdq3hBXFdax6TGsGHkNMa/WYXwbT+UU5IWTgynBfTPm5Y6Bo9OHOhhKKv0NvvmT08NC5dfJyRAxcvRfze4g3Hvl+xXw5ouvkid97FH94TvbKUH2T0zJwtay83uQyjnailDPwbb0FjGvzR49ifRFTRJSLxDSl513X1e+/W1fPwP/tdwvx6KsbwxQUi8VYBU/NRmiexFTRnnP9jqtYkt5E3IDrJzGB+IibT0RyCUVk0ly+E9+drn2TcB4RfcpoT03JYcJAglNErdBUBjzJDzwuobAwCGNUi4v9Jo0BEE9H6jokRIxFD7lc7hIAuPS3C/G6ZlEAE2LJn1Iy8PZc+akKUa9WWXVHwHh0Rq9YYppnGT9nRpFQOHYTDXi+WHb0wDXw/kLJGCTDTxFn4F7YcasdSeKitmUaj5gXXHRhVNfFBOK5yssacM21RTdCk3Hl97m5ZwC7jOIGPDrXxh79SjWqXJRxSOseB/iBQt39RWQcB6b8Yd39hbA+8Ots3ZHHK+u6jaH0HLrJWcA3pKNyLnoHSpL23hYa8FKwX5KEolwrYM484K6gkVBEA66f/9C3+x4liG+rISpbhTgK+I/bXsSyt1PHP6ZGUxlw8YWqfuBqBSp6DIde9SAAP8/xcXtPAOCvbCLCdShkrPkKGTgAPLDk7bL7zJg4OnZ+ID6Mi9bZlLePTpFkSHIjVKQabsDVe+cr/qjPkj9nMSmR6+oNuKiBi2HHh08bJ+1XDD1jOAMvxpiNWC4g7oVS8hhag7Kqxj+J4ajasUoWxcAg/uxfWi0/Kw41AlhnwMWAEF2H5nkRMzclZeP33t1fDD1ExDq5uUefJlkvoeivwZjva869sUzl4KyXV1fuXy3mGdJ54/RrpCHAr1/tuQx68yWJeIXujWF2wPg5nXIMPG+WUERJRpdAzPMY1nfHo3zV9VG7NA4NOojtfaDolc0vUw2ayoAnhdInMaFRLRkcvPtYAMAhwd/wOBIYuPDS07qpPR6kahWxdyDXcPCKo07KqZ0E945Rh91p0nyKE45qgIqYQ1kE9xlXGXgu1Kv97zoNnGNih7xqCtcg/+3d++AXHzsyKp8yOdiX97STd/weAH3+axMDTzTgnoev3/0KXl7TFd6PCJGB8+coDpPFuqUaQ530IBplXT0qCQxc7Ty5JHXw7pGf8y7tyRJKWT9wjSHcp9Ovo29t60MmiIfQoVCKAsN45/bWVl+ye21jT7if7miThNKScdCec9GbL0rHqQxc16mYJjEHFL1dNJ78nYkGXOc++9KabTjv5/EcMKoBN0l/KlQ9PWmlqWrRVAacM5HRLZmY072pAgJ+ljXXIcz7won49SeOln5zHb0GXq1kddCUDnzpdDmgKJyEUhqz2ri/ee9SLHxza8yzJZUB53KHF/dC6WjT+/iWPH+eoE2RaLgBF/3ATRq4OjrggRSuQ1Lu5jDCMZzELBqXPWMGBs4YJA38ew+8GuqySQZ80/Y8bnpyJc65/gkA8ca7NphDYIxpNXzRkKjEgbNtsfqJ/ss6FipKKKqnxbv3n4hV15yF6RMiEjAuqPeiG+Emgc0xxrChux87BoraSEzRcPDyThI63iQvlEIwjwREncPEDj+lqygx6TVwAwPPOmhvyWBHviQdFxrwggeH9G2a76+GvatyjczAuQFPnsQ0ZWxUA9bSrjCvjrirWZCmHJrKgE/saMXjXzoJC796Smx4leQGxTXkvXYdFeqJ4XEOhUPDaldxUaFW5oiBJ2vgALDwja1xHdPQc4sVQszhoV4nmYEztCmTRXzSR5zQU90I1X15e+KVnUhuwFG+8SC4I1/ShrI75HcYjDFtJkDOWLf3F3D9vBU4+8dPYOWmHVoD/sl37Q0A4WK1HOK+RBGbNXUCIttT8+1wIyOyOzGCUMfWPBYZQ9XTgr978d2Oac0i6xIKJQ+begZw85Mr8eLqbaEUVPIYjv72XJz386c1uVBIIjvcmI0TAncyCRJKoeSFxpA/Hl1b0/En0QtD7OhbM26ggcvvl7/bgaJnXNQjlFCUC3LXSA6ZgVN43fC+KkgupTJw08hRhcrA0y5CUwmayoADwB7j25HLOLEemFfA0w/aLXZMkobsEgQ/8MHPFBPFh5NcU567bIMcHakZXo9uyWij6XQQHwE/hLNqEaLfsoi+fAn5kiflWAZECSXOwFVWlXEJq645C7/859kARANO2G1slHyfa44RAy9p5YWs64AxFrho+ttETxfO0kQd8qTvPartDA6c4ssQYqTc9CvuxXOroqDhaePbsTHQk03RcpIBV6U7ihtwkYF/5/5l+MsLa6RjxBV51CRfnsaAj25xkXUdFIoePvvHF3D13a9g7rIN+MTxewXH+PstXdcdk/QyDiGXiZd/bFv0zt0ECSVf8qL1RcPgOP95/e0z7wz300ZiCqy4c0xUF1qyXEKR3U55OxkoloySaCihCNfbfZy4ck989MY7A/EdVZKbRM1EWo6B86KtECQmoPwiNNWg6Qw4h8pK+UP7+ceOxBdPkyWM9hazBOEa0snqsJeibetAiA9H+dDtjgVr8CNhVR2dAevNl2KGxFyZo+0iW1b9VjsMDJz7Xe8qrHICREzBYwyL3+rCS6u3hdKCqmvyffkoR4xy3XfiGLz036filAMnhQ0myrNd0noCZF3Hd3kMrvO1sw/EDUHnAOgNOKDPt2zSHMXnPm18eyhHmBxoxKevnpOPBMV5BLWh8qx4HOLyeCoD54+kRTLgGd+Al+SJMD6fI056P/SKnKfDT2YVnzvqaM2EbSbjOFoGDfhrpHJNel1XPy759Xy8vnEH3jdrSthB8uvEjhWMpEiioknMokRWZAZuqPNOnIHvv9uY8HMYVS268IWjHVFC8YJyA0dPH48T9zenuo5JKGUYOG8Td78k58IZ8RKKCDW/g2jk1Mo0KsENT17QIZ4dTUS5EFogYODKfmIj/dHc5XhtYw/6DQy0u78QMyTcSEwc04LvnndodC1hHzHoRtXaTRo4d4ca1y4z8GzA2Eoew1V3LQEQ+ZqrlZd3pKOCTpJXdv4OxrZlkRFcNXkDUyMxxXv15RP/Ou05VzJmGddBznXCEHUOHQNPM2k0bXw7NvUMSLp0EtRGyO9TZNK6HB4imCihZFQDHjBwN9o+mksonuzjz+/PNFkIxEPpuXFpy7nh5yQHAFFC6eor4MFX1mP7QDFWZ8qF4ovGsyXjYFSLix0KA+fS00DBC72eVJCGgYsTwdqVpcIVfkQJxY/Q9Rhw3D4TEtdhVV2Py8FkqOsxidlUuVBEqJVONuDyvqMSGbgcidmWdY2zzOXCjU37qQEJPHLyQCWjGuAnvZqgSBr83k7afyJOO2g3fOmORQBkrV1cykxlth0GCUVdoYgj1MA95kdavrE1/E01crxzGdXCGXghKJtcfl6mkpALRTcHwBk4fwdtOTfmD9yWc+MMXEOfTf7PIvYY346BooeegWIsi0vO9YNsxO2xSMzg3eRcB0S+cS43VPaEd9SWUyQUIcCHQ5RQRO2W75O08IfryGXmx/COcaDoBZ2m/nhxElOEKruVN+CixOT4DFyZdI0YeMnMwDWRmO2iNBI8H5EcjW7Jxspw+/zVeCOI4XAdivmVi1A18HJoyTjYrtleDwbetAZc7c3kxDryy08KhHGEbIT5IKhFZ8Bn7TEuZDp8ok0HCv+LoMvpAPj5Ejj2nNCOrTvyeHjp+th+vDLnMo5UccXbFPVqVQoyzQFsDQz4jEmjpe1ZQUIpl6WOG3s+ytm6wzesoqQge/r4f9VhqXi+osdwzX1+2HZ7LhOr+G3ZuAHXyjEJrIqjM5CP7lv8Nh5YLPv0t+Vc5PvkFLktrvwuRYPikE8GVFlEfR9+LhQTA/f/ygY8G0ooIgPn+5iWfePl0zLwrBukZ/YzSprGHnmBgYuYKcgW/DpJEDvh1ozra+CFktRxiJGYplEB3ypKKKKXFn+2nBz88COzQolFfC+PL98Uzhe4hlQCHKa6aoIuAAmwEoqEGANPSC6f9HIkBl70tPkXTjlwEm6/9LhouGx4QQDw1bMPjFVm3eITajlPOWCSNmUtEDWO0a0ZJf+EyMCDv17cC4UnD1LBNfBZe4zDk1fMwfH7+sFOoh84r7xfPftA7Tl458LnGTYqy9YBsgHnf3XZHQGfNS9d142Hl/oLPPGwaxHtOVdaaUk8rwhTigARewY5cr50xyLMXSYvKsU7IYmBZ+Rz8vvPuE5oXFQGHouA9cyTmPxa4jk62jLYdXQOT6zYhLeFQBNujE3L/fnlc6SRCC9/Wy4TXsPEdgE9A3/yijk4VXEWKGfAJQklYOAq6xcNuEnS4IfIEopcvwte1OmcNDNKO6sLzQcCL5yEzt7kDmmCqd4NixcKEd1ERBuIaLGw7SoiektZ4GFIoQ6PxfojVqZfChNgOoiRmIWSp136qz3nhomv+DHf/cChsf2u/dBhmD19fMwLRdcpAP5wuzXr4Jkr34MrzphpLCNnl2OEiSdAvmdPmMRUteVJHa3462XHx87LDfi49hx2H9cWVmJxErOnv4iZu43BaQdN0pZNlBCAyDCLnVZGMOD8Xkw+t+rIqjUbN+C5jBMb1uoMuDjRLc4diDhwSodxbkPnf682Qn7/WSElq2rAN3TLEXgeixgiH96Hv2kklEkdrfjcyfthU09eMnomBs7dJwHAceQyiwycl1OXRpXDn8Qs7w5Xbn5IklAyrlbW5AZ25aYdUn4fEVEu/2jbKOU9FUvR8xXLahoJ+5GoyeWvBEYGPkxuhLcAOF2z/TpxgYfaFqs81IYuDqF5ZepozeCUA/WGh0OMxMyXPO1L5o2Un5cIeNd+8VlrU67iJCaYcx3sNrbV6OsNRA10TGtWllCEfSQJJZYMy8GhU+UIVMCfxMw4FDYAXkxuGIolhu39GK9+aQAAHvxJREFURYxuyUgS1X6C5MK388rJgztE46fTwPn3q845ELdfepxUVhHtwmQbR8YlowEXO2zxuc+ZKS8AEJ0/E+bJUaEbOamjOSdk4BSmGVU77PXb5fBsj0V51VUffV4XRVY8YVQupjkDAgNXGP6pQmerppMVNfBcyMDjde9/3n8I9hjfhrVdffjR3OXydbUr5cQ2SVAnMXVr2qZJwsk7OEdi4PK5lr29Hd+9/1W/rJp7V+GWYeCVwnSdpGDDalG21IyxvwOo+2o7lUJtSKLvsy64Iuk8oT5b9PSNVgkeMIUn80qlVuak/MHiMnHnHDZFuw8fgnckSiiBAfdYzA/cdUgbKVcosYDVyxND3IiWGEPPQBFjWjMSw/rwUdNi53Ic8r1D+pI1cHV0kFV0ffW5+qMftUN0YkYrSg5F0n4cSbEAU3dp024PJRShyOpzDCUUxwnfuzpUVw2Txxj6CiXkXCdWR/l9iNfJuI52YjTjEhxCLCBGZIB+NsL4M2kTDLjPwOVz773rKGRdR5sqQlcWXf0SIRKjloyrbZvlEoMBkVQo7qu+2+sfiToc0Wiazp8pM4lZKeqhdZswmCt9iogWBRLLLqadiOgSIppPRPM3btw4iMvJUCeoRMPAK5NJexbhOCIDZ1qdjMs13Hi6FE+SD/hasn99ebtppXZAbgw/Pv/wmA87IBrwbIKEwv/6GrhJ74tfP3pGvEPgK7R4nm/AR7dmpU5o19H6lXBymWhxArGBZhwHW3bk8f0HX41NNmYdR+qUVD22LefGjEPWcWKSSTE04NF9m4w5x/uP2D0stw5i/eFBMyrCZ+aKEkp03OdOnhE7xmMM/YUSWrKOVMYPHDEVV597kPY6ujJmgolTtTMT65TqRsh/a8tGIxudBu4GHbIOumep08Dfsc+EcAQsRapmZQb+hVP3w4/PPzxMOAcAn3x3JAOJCBeDFso8SjHgLxgSkZkMuOtEJGLa+Hacfehk4b60h2BMAiGoh1RiQrVX+hmAfQDMArAOwPdNOzLGbmCMzWaMze7sNDvLV4qDpnTg4++YHr50nYSSJoufuir9mJa410UooTicYccr9wtfPQV7BvkrVMadNDRUG+Yu7XHjyBlWR1tGMWbRZ34Pa7b24R/re7RDVI5vvu/g8LM4KSdq/G7gUcElFLEim/zq+b04FDciAPDjR1bEc51nSGHgqoQSv5Y6kQhEQ+uM4Vxq4/3n4/bEtR+aBcCsWXIZiIHha+cciFXXnBXbh1/CD4bhk9zRdaeMi7N7z/MNuGhEAeD7HzoMk8fqRwNaA+7611Q1cImBKwacf27PueF8j84gkyGPuMljQ9101qGTcauQd2isEIugMvDWrItzDpsizWt9/B3TY9cAokhLsdNX27kuDTAvu3571EY/eORUXH/BEdr9RBw4pQOrrjkLR+81Pvabbh6tXqjqSoyx9YyxEmPMA/BLAEeXO6bWcBzCVe89CPtM9I2mGMixJQiNPmKacWAQQtRnCyVP67EhMm//mPhQX2QEaj1Jig9ROwLVLxiINE41JF5XH+cHPts6o/TA596Fp6+cg/OPjiQQ8fqiRMSXmusZKPg6rWjADR2jOEEmdjRiw1EDbjKOLKHoXAZV6DRb/g7Fc6mjpLMOiZiVFJ1oYEw6CUUF106zLgleKFGZx2s65FWbe/Hsyi1oy7nIJgy33zljV1x+yn7GMvpZBONeKKIBcRw5Hzh/vmIgz9j2bGyUaJIJTfM5KmnpaPWjR8OOVTiuJetII8Ro5Ff+nZRY/D2PNkQaqyMLE6FzhRGdWv/Kdai6NBcNz8CJaLLw9Z8ALDbtW2/wly5KKDxX9JwD9BNXIrihKpT89KZqWLm/T8Asg6flUJyFyP7Z8m9JixykMVh9oQGXK6B4mYxD2E3IMLetL550fv/dxmDy2DbJ8OcUvRSIDPhA0UN/wQsYeHSQSZ6JjINcTsmAqxq4S0bWrB7LfY9FI8LvmXtKSGxeeUc/ufCIMGRaZ9RUqPehQ8jAXSd87+LzUZOnAcAtT63Cyk070JpxE6PzfnPxMfj0e2YYy5gJVtKJM3Bh5EEkyY1iJ8vfxNi2bHwxZAMDNxknk+suN7gZ1wknkttybszNVPwLmN+JLqGWbkT4swuPwIpvy85xB0zuwLUfOiy2b0ZwJVbv+d7PnID7P/fOWNAd308399lQGjgR/QHA0wD2J6I1RHQxgO8S0ctEtAjASQD+o87lNII3fpHZ/fuJ++CnFx6BEzWeIipcx9cQH3t1IxgDdtdMaPHGLiaTV420aCtUZqxz0eL7qC9brNhfPn0m7vn0CaFLlBpRKbKeSR2t0jCV+67qOgSiSK/NCYaMNyCenY57erRlXYlfmXx0IwOueI1IDDzuIeNIBtz/fMDkDvzpk5F3yktfOxX/F7hCio3s9IN9f2SuvSfJMUDUSaXxTogkFDP4+UQXT/EdioFQxyjD7VbNBK0JJjnDN+CqBq5KKMIzCb1QMuFx49qzsXt0HP1zyRnkJtXDgteRcHLZIfz0wiPwl39/Bzpas9IzcoR6F13HYMA1Iy2dS6KuHQPA+4+Yqi27iYGPa89h5m4dMRLGnylvgwcJeWEayoAzxs5njE1mjGUZY1MZYzcyxj7GGDuEMXYoY+y9jLF1Q1FYHXhuYlFeGNOaxZmHTC47Mw5ERvlffz0fACQWq+6jYwrqPoCOgUefzzzENzjc2MYkFKFi/9uJ+4QLUQBxv2Rxki3r6jPKmXKJ8/KKlU1M1ek4hP6A1eYyjnR/phB1cYJMhPi84qH4cmZJLntMHtsq6Ytj26NGL0ojfFjcrzHgunLyCa4jp0fyWjkDngT+zDtas9pAHtH979Nz5AnNloyTerit2y/rONoowZwy/6BLh9uSccJnNq4tPkpQtfPoeP27V6sery+eIHm0Zl0cHsiasoQSXVMtp4pSaMCjbTppRFzqrxwyCQacI27A5f3OPGQyzj96DwD1yTpowtBdqU74xPF74TsfOAQfmr1HVcer9WSSxoBzAxO6E2rqsCnEHZBfPpcQQgOuSigao3HPp0/AV88+MNYx3HiR6PPsSO6D3FPE5Injagx4NCHnN3rewLOuI92TMdLMJKEIBxc9T6rgWWVB3WxCJ6m7Po8A1RlwnQE6LPCHP36fXcNtpgYXPrsECs7fZ0dbJjTmogYtjppiUZeMpU5wpFtdXpfwiUh+Bm4wWvzUSfvizn9/B/YY3449J7TDcUhm4NrFkP3znLDvrqH0YPRxVt0rg2NNa39KbqaCJ090H/5nNS8QlzpkSS9ex035f3QQ18U11QV1HiuWF56igMCkSO1ao+kNeMZ18OGjpqXyIdVBdeDnjF6E6H0ifpf2ScjFwhhwybv2xudP2Q//+s69kXEIZwQTaqp+qWN9B+8+FhefsFds+96do3Hpu/cB4DeQt7v8gJFPnbQvfv7RI43n8+8F4XEcoreNQ6IBlyUjo4QS6qvy72LlLwa+5xxZVw715uw6Kbxb1HS5O9dTKzb790DJBvzHFxyBp66Yk7g8HwdPkpTkBsplJpGBi9d1HMLeu45CS8bBfhPlgKF8iaVKuMWh3o8u14t6Nl5Xv3Da/jhi2i74l3dMx4P/8S4A0dzK2La4hCJ6oew5oT30pjE9K/W6oQaumcQEoJVQdG34kctPlOaleD8jdVKa4yphwRknMr6mDrUcA3cdUZ6K7u3nHy3v0TIYNG0yq1pBfPf+MmBxAx4ycGESM+k86q+MMfznmQeE31d8+0zMW7YBP8NrsZVE0gQf6cqWzTjhwgUfmr0HNu3ww7dNE3FqCLx/D4IG7kSsNpeRGbjJ6OQMEkq+FGm0fYUSxrRmsSnwFOJsX72fRAYuuBFyN8P7g8WldZNjIka3ZGJDbtNwPc276AldPLOx5b74KOiuT5+AlowTqxeFoleRx4I6fyDe30n7d2LeqxvLyoaOQ2hx/Pvi8wbj2rOxUYbrUJhTZ1JHa/hOTQZOvS7vSEsa905AeU8aLxSOse1Z7Da2JbZqklOmo056Dg9//l1Y9vZ2fOr3L/jnEhh40gLQIngd5J276zj48ukzsUt7FuccNgU/ffQ1AMDpB09GPTHiDbiYqGaX9qy20avat851KCkbos6NcJ9OPxz9bWUV7CT/bR04e8k6/jC5r1DCtAntmDyuFeccNgWffU88kEQso6qXAv79uUThs8m5igZuYOAtIQORq5WYzW7HQFHKT5515UAe/jGJgWcE1z1VlkgTvGUqd2x7cO4kN0IxSpaX2CHCzR8/KsyCxzsMVaYolLyKckSrScrEZzQ6kAwqGYfy0d9YTb54lwivb/TTrR4wuSM0uGYJRf4epk5gvKxm7yL+k6nTPmr6eCx+y8/c6WncCCsdfe87cYyUOC4juBKbXgd/d3zfnJKV0iX/OX7xtJl4XVmJp54Y8QZclDDUJPW5jIO8kNrSFCqvIkkD5zDNkldqgMJFdR3CF4Qozqzr4MfnH162jCID5MV2HQetOTd0Rcy6Mnssx8BVX3oxGVLPQAnThZWNsq4jNe4orN9s2Lg2m3Pjk4DtORc/Pv9w7N1ZfvUktdyx7W56Bq6mJDhJk3tFZYX5kpfoB65CrUailMc7ibQ56wHgln85Gn947k3tJKDjREuCHTilA6UyEoN6XXUl+ySpyFFGLir46PXmJ1dpQ+kB4IyDd4NDhHtfTudPkVOkQx2zF8Gv25Z1sX2gGAsmk0eR8jO65F17G9M1DBYj3oCLQRBq7uvWwICrw/pyPX7cgMf3cR3C/553qLQclH/Nygy4mmir0uN0xst1gClj2/DECj8PRlbxQjHl2eaNQu0IRea4Y6AoaYS+90y0L7+PRA08uE4248QMYGvGNeaUMaFc/uYkN0KugY9qidwIk5IWORTVh0LRS+1GWA7hsnkVnO7YvSfg2CCSWb1H1yF84IipuGPBGkwZ2xqmCTaNVmIGXEkfnLTiDf/N1K6yrhMmHNMlswKAn330SCxZ25XagIteMGJCO1M5w0nOwICbFrf2P8vHivJprTHiDbjIwMcq7lQtWRfoL0r+30AKAx4LpdebgA9qPGfSZCy7/oLDMW18e0Vlil1HI6HwUrqOgynjIm8cvtoMh4lN8UY7ThmSiwy8r1CS2LXKwMP7STBsfKIzq2Hg1WR8MzJwbsATNBQuoYzKRekGTItQA8ECIkLunTSLTqRCcMn9lMU50kLnhfKdDxyKb5x7MIgoklBMGoNyy7zTDg1jwvvkOdmT5CT+THUSCkclcpTYaWfcaBLT9Dp4x8E7MH4t3tblfD7WjXDIICYCulRJoMNd2tRAHj4U5uxJzTWtvr9jhSQ9tcDZh07BoVP9xFkZRd5JCye8BwMDF3J45DKUyoBzY6bmc1FXdJH9vmUGTpp9VOQECaUW6wyqRun/vXMv7DtxNPZIMez995N8L6DOMS1RY05k4NFv33zfQXAcwnlHTsVvLh5cNgoepXrlGdWxPZ0XiutQKOmVm8RUb1ldwCNppMHZdVLfOyGYEOb+3bpOMqnOqJBz9ThRx1BGQukOlgxUFwpPYuD1hGXgwRD49kuPw+zpcqTc2sAtb0bg/sXrLq+Li68+DUB8CM4b8piWDJ66ck4sh0kt4VRtwP2/EgMP9UUHU8ZFRtjXwMtLKHw0M67dzMD988seBHoNvDwDzwnBKIMBfwaHTR2L73/oMOw7cQy+chawrsv3EEqSUC48Zk9ceMyeQdlRtuzcj/mLp+0feih874Px8O5Kcfw+u2LZN06v2IvJBPUeskHAVVo/cL50XpS7xGzV+DmTPEfmzJyIn1xwRJjdUHe6qhm4oIGbyAk38HwhEh6QxL1QxM4jaQRWa4x4A/7u/Tox/42t2GOXduM+RwVRe+pkpkk7Dd8lxRNQ1Rq8vlWqHOgmMUOXKJIjUrOuI6cKMFwsCgyRGfgpB07CQ69Ea31Kod2Ocu4yE1q8PLzsugV3Rfzgw7O0GQFFiFr3voKvdqVD4TRyVkkZitcKuUw8t3gl0OVCUdGmWR2JI27AlUlMzTP57cXHpPaDJyKcJaR51T3jSnzq24Xw+6QYDg61Ez9wipwbJWkSs54Y8Qb8spP2xQXHTMMEIVjghx+Zhd89+yauOGMmuvsKIeNLCuQRwevAYPrhtJNbLk+qU2Gvz+cV9QycpGAb3w+8/Pl7hcAQER88ciqmTxiFD/3iaQByBc8oQUJRWHUKL5SMgxP378QJ++4aTriqeN/hu5ctd6R1y9u50UmzUoyINO+ipUZMOTzfIKP/1GAl3ePfbWwrJo6Jx0kA8Yn7UEJJ8K8+YcausW1poZdQ0htOsTwZl/CDDx+OXz3xeihNquBzBNdfcDh686XY/UidgJVQhg6OQ5LxBoBzZ+2Oc2fFG76aVtYEbpDSGD0dnvvKe1IHeKirBaUFX9pMex0iyZc7pzBwE3hkn5pciIgkWWX8qBwuO2kf/GTeazFGl8YPPCtIKFnXwdfPPQhzvv9Y+QIaEBpqxYhVwuiA8v7MIurBwAeDGAPX3MPtlx5nZPlqVY8YuPl8g4GuvovupZXAdQjTJrTj6+cebNyHa+DH7T0hZi/4OTgsA29QRKHmyfsNtqpOHFNZIh6gct2Nz7rrvFAAKK5+cQZ+9XsPwuHTZLbyw4/Mwk8ffU2bQ1nMIT6xowUXHD0Nl5+yf6whptLAlShS3aIPlcC05F2lDVHnkWBCrbRqDvV5zdxtDJa9vb3q8+lGEao0lrS/6kZYK3dJjqRguko7szTvK8n7BbAMvCmguhOW228o5jKclJ2KCt7AtAvUQja4vPG9e79OXHCMvxjERZoVU2ZPH4+bPh5foQQAJglD787RLVJKWxFixJsJIgMHZD1zMEjDQpOQZhKToxoGPqYlE6ZLKIe7PnVCbPGMSlCtVxPgLw33H6f4EcBiPvBaQsfAR7dksHfnKHzx1PiyhElItRanZq1S0zksA29QJOVCEVELDTwteJkqlWt0WeJEAyYycG4oxSWyKoXYgDsNOiogLFqb6Afu/xYuDzZINsslH+6PzME7LtMaoCoq8cmvhoE/8oUT0dWXx8nX/r3svrmMg1wFXsKD7bzE3b8vLJqQNImpw6/+eTamTTA7FHCY1vF85PITU11HOleK0cHv/9+xuGPBmpj7IH9uYnlqrBYloqwBJ6KbAJwNYANj7OBg23gAtwGYDmAVgA8xxrbWr5iNgbQNVPUXrycqGbaLKHhRru/YOQlKytfaMookA25KfiSCG3meq4R3DrrVlNJg787R+MXHjsTx+8qTakSE733wMBw9XT+qUMFLnOZxVZrzBvCfW9KzqyUq92rSH1BKMaIScXLgJlgOtdTU07Sdg3cfK+XmVyGOCIai3YfXTbHPLQBOV7ZdAWAuY2wGgLnB950ekc918n5D+P5Cba7S+syZgzzhE9EwsRLW2oAnGVqWwm+YBwa1COX6/b8eg3s+fULVZTrtoN20OUHOO3JqKkYIVPbeK8nVMhRQIzFrZYQi/+oaSyg1KF8lklc5DCXrFlGWgTPG/k5E05XN5wI4Mfh8K4BHAXy5huVqSLgpGfhQ9sBhEp4qa9BMJRcLEJ/UqxXbuf3S4zB36YZE+cDTDElVcAMudizv2Ld6l7Ragb/3pEWsOaodLTQbdEug1QK1ON/onD+nMJjOIJJLh8eCV9stTuLLqAV/y68evBOAG0ld+k1pvyHUwDlxqrYSipkBK/V3rhRHTR+PK86YmbhPOKJINOBRnvJGAi9yvZ9jPVCrIqsufJVKKGlRi9NdcKw/Ia+mJG4m1H0Sk4guAXAJAEybNq3el6sreKVJcqcCot54KIh4qUoJ5dEvnBgzgNzwDKUEpCJaddy8D4+WPGJPfdDFcIG/96QVfL71TweHicgaCZ87eQa+/OeXB3WO7553KI7ccxdp2zfOPRj/c9+yin2zy6EWo9wvnzYTn5kzQ+uG+oMPz4otoNGIqNaAryeiyYyxdUQ0GcAG046MsRsA3AAAs2fPbvwnkgC+wEFaBj4UHLycf6oJIvNuJIRD7oQGesKMXTHvCydirwa7B+7NkjSc5nlTBgu+lNz3P3hYbKHoavDho6bhw0dNw/Qr7q36HLp1aT84ew9t1s1GgOOQ5C4rIk0ELzD8o61qDfhdAC4CcE3w9681K1EDw5SJbDjhhax58J0FZ47DSMCFSdnkUjSa8QaAn1x4BG6fv6bqlK5pseTq08Ln84Ejp9b1WhbpMFyj1jRuhH+AP2G5KxGtAfDf8A33n4joYgBvAvhgPQvZKOjqCwx4GQbOO+WheKlpGGszwWsAGadaTB7bhs8YlrCrJUys0WLkIY0XyvmGn95T47I0PLoDA15OQgm15HoXCNW7EeqgauB/vOTYcKX7euOuTx2P9lwGP523AkBt3MQsLIYLe4xvw0n719+3w3blFSCtAecYChvEXfJqmbaWa7i1XogiCTwLXKlKTd/CopHw+JfmDMl1rAGvALOnj8dLa7qwe5n80kleCLXGeUdOxZYdeVx8wl6DPlcjzDA3s4SyM+D4fSfgyRWbh7sYqbFPgwVEDTWsAa8AV5wxExcdNx0TO5KzBUYSSv2tUNZ1cNlJ+9bkXIdOHYs7FqwZVg+VtJOYFvXBrz9xTE28WoYCT14xp2EcCobLG6Ux7r5JkHWdVGHVfJLz3MMrWx19uPGxY/fEcXtPwIxJ8ejMoQKzBnxY4TrUNPJVuZHwUOD8o6fhqdc2193zyARrwOuA0S0ZLL76tEFnyRtqENGwGm9ASA3QHDbEYoTjnMOm4JzDho+oWQNeJ+gSI1mURxiJaS24hUVZNG8SAIudElZCsbBID2vALRoKaXKhWFhY+LAG3KKhEGng1oJbWJSDNeAWDYU06WQtLCx8WANu0VBgVkKxsEgNa8AtGgo2kMfCIj2sAbdoKMyZ6ScAasRFDywsGg3WWdmioXDxCXvhA0dMxS6jklc9srCwsAzcosFARNZ4W1ikhDXgFhYWFk2KQUkoRLQKwHYAJQBFxtjsWhTKwsLCwqI8aqGBn8QY21SD81hYWFhYVAAroVhYWFg0KQbLwBmAB4mIAfgFY+wGdQciugTAJcHXHiJ6tcpr7QpgpDF9e88jA/aeRwYGc8976jYSG8RSEkQ0hTG2logmAngIwKcZY3+v+oTJ15o/0jR2e88jA/aeRwbqcc+DklAYY2uDvxsA/AXA0bUolIWFhYVFeVRtwIloFBGN4Z8BnApgca0KZmFhYWGRjMFo4JMA/IX8nBUZAL9njN1fk1LpEdPXRwDsPY8M2HseGaj5PQ9KA7ewsLCwGD5YN0ILCwuLJoU14BYWFhZNiqYw4ER0OhG9SkQriOiK4S5PrUBENxHRBiJaLGwbT0QPEdHy4O8uwXYioh8Fz2ARER0xfCWvDkS0BxHNI6KlRLSEiD4bbN9p7xkAiKiViJ4jopeC+7462L4XET0b3PdtRJQLtrcE31cEv08fzvJXCyJyiegFIron+L5T3y/gpxchopeJ6EUimh9sq1v9bngDTkQugJ8AOAPAgQDOJ6IDh7dUNcMtAE5Xtl0BYC5jbAaAucF3wL//GcG/SwD8bIjKWEsUAVzOGDsAwLEALgve5c58zwAwAGAOY+wwALMAnE5ExwL4DoDrgvveCuDiYP+LAWxljO0L4Lpgv2bEZwEsFb7v7PfLcRJjbJbg812/+s0Ya+h/AI4D8IDw/UoAVw53uWp4f9MBLBa+vwpgcvB5MoBXg8+/AHC+br9m/QfgrwBOGWH33A5gIYBj4EflZYLtYT0H8ACA44LPmWA/Gu6yV3ifUwNjNQfAPQBoZ75f4b5XAdhV2Va3+t3wDBzA7gBWC9/XBNt2VkxijK0DgODvxGD7TvUcgmHy4QCexQi450BOeBHABvhRy68B2MYYKwa7iPcW3nfwexeACUNb4kHjBwC+BMALvk/Azn2/HDy9yIIgjQhQx/rdDCvy6BZHHIm+jzvNcyCi0QD+DOBzjLFuMq9/udPcM2OsBGAWEY2DH7V8gG634G9T3zcRnQ1gA2NsARGdyDdrdt0p7lfB8UxIL0JEyxL2HfR9NwMDXwNgD+H7VABrh6ksQ4H1RDQZAIK/G4LtO8VzIKIsfOP9O8bYncHmnfqeRTDGtgF4FP4cwDgi4iRKvLfwvoPfxwLYMrQlHRSOB/DeYL2AP8KXUX6Anfd+QzB9epG61e9mMODPA5gRzGDnAHwEwF3DXKZ64i4AFwWfL4KvE/Pt/xzMXB8LoIsPy5oF5FPtGwEsZYxdK/y0094zABBRZ8C8QURtAE6GP7k3D8B5wW7qffPncR6AR1ggkjYDGGNXMsamMsamw2+vjzDGLsROer8cZE4vUr/6Pdyif8qJgTMB/AO+bviV4S5PDe/rDwDWASjA740vhq/9zQWwPPg7PtiX4HvjvAbgZQCzh7v8VdzvCfCHiIsAvBj8O3NnvufgPg4F8EJw34sBfC3YvjeA5wCsAHA7gJZge2vwfUXw+97DfQ+DuPcTAdwzEu43uL+Xgn9LuK2qZ/22ofQWFhYWTYpmkFAsLCwsLDSwBtzCwsKiSWENuIWFhUWTwhpwCwsLiyaFNeAWFhYWTQprwC0sLCyaFNaAW1hYWDQp/j8oFK+geDr0FQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses)\n",
    "plt.ylim(5,50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用autograd实现的线性回归最大的不同点就在于autograd不需要计算反向传播，可以自动计算微分。这点不单是在深度学习，在许多机器学习的问题中都很有用。另外需要注意的是在每次反向传播之前要记得先把梯度清零。\n",
    "\n",
    "本章主要介绍了PyTorch中两个基础底层的数据结构：Tensor和autograd中的Variable。Tensor是一个类似Numpy数组的高效多维数值运算数据结构，有着和Numpy相类似的接口，并提供简单易用的GPU加速。Variable是autograd封装了Tensor并提供自动求导技术的，具有和Tensor几乎一样的接口。`autograd`是PyTorch的自动微分引擎，采用动态计算图技术，能够快速高效的计算导数。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
